ATLAS 3.8.2 errata

[Home] [Docs] [FAQ] [Errata] [Software] [Install] [Support] [Lists] [Developer home] [Timings]

ATLAS v3.8.2 released 06/06/08

ATLAS errors:

Known issues:

Noteworthy compiler issues:

System problems & user tips:

[Home] [Docs] [FAQ] [Errata] [Software] [Support] [Lists] [Developer home] [Timings]

SourceForge Logo ICL Logo

ATLAS configure mis-identifies your new system as an older system (eg., Core2-Xeon detected as PIII)

In the original x86 ISA, when using CPUID to detect family and model, we were advised to only add in the extended bits for certain base bits. Intel and AMD now say to always add them in, and have reused the base bits for newer architectures. This means that 3.8.x (which uses the original CPUID instructions) will sometimes detect a modern machine as some older machine (for instance my Xeon E5420 was detected as a Pentium III). To fix this, simply comment out lines 95 and 99 of ATLAS/CONFIG/src/backend/archinfo_x86.c. So, change line 95 from:
   if (*family == 0xf || *family == 0) /* extended family is added in */
/* if (*family == 0xf || *family == 0)*/ /* extended family is added in */
and change line 99 from
   if (*model == 0xf)                  /* extended model is concatenated */
/* if (*model == 0xf)*/                /* extended model is concatenated */
Essentially, all the Core2-based systems are treated the same by ATLAS. So, to get to use the architectural defaults on Core2-based XEONs, change line 297 from:
      case 15:
      case 15: ; case 23:
Finally, to enable better P4E identification, change line 313 from:
      case 4:
      case 4: ; case 6:

Threading doesn't work for Windows/cygwin

ATLAS presently will not autodetect the number of processors under Windows, so you must pass the '-t [# of processors]' flag to configure to force threading to be enabled (eg., I add -t 4 to my Core2Quad configure command).

The second problem is that cygwin's pthreads does not support setting the scope, and ATLAS requires that it works. To fix this, edit the file ATLAS/src/pthreads/misc/ATL_thread_init.c and change line 99 from:

   ATL_assert(!pthread_attr_setscope(ATTR, PTHREAD_SCOPE_SYSTEM));
   pthread_attr_setscope(ATTR, PTHREAD_SCOPE_SYSTEM);

ATLAS threaded performance dies for large problems

On some platforms, if you time very large problems, you'll see that ATLAS's threaded library does well, and then suddenly drops to below serial performance. This is probably an error in how we recover from lack of memory in the threaded case, but in the meantime the easy fix is to increase the total amount of memory ATLAS is allowed to allocate. To do this, edit the file ATLAS/include/atlas_lvl3.h, and pump up the macro ATL_MaxMalloc, which is the maximal size (in bytes) ATLAS is allowed to allocate. It is presently set to be 64MB; make it as large as you think you can afford.

x86/Solaris install dies trying to assemble GEMM kernel files

You can thank those geniuses at gnu, who appear to classify division not working in Solaris assembly as something the end-user should fix, for this. To work around this excellent gcc quirk, make the following changes (in addition to the hundreds I already made):

Change line 50 of ATL_dmm6x1x72_sse2.c from:

#if MB == 0
#if MB == 0 || defined(ATL_OS_SunOS)  /* retarded gcc on SunOS has no div */
Change line 100 of ATL_dmm8x1x120_sse2.c from:
#if MB == 0
#if MB == 0 || defined(ATL_OS_SunOS)  /* retarded gcc on SunOS has no div */
Change line 108 of ATL_smm10x1x120_sse.c from:
#if MB == 0
#if MB == 0 || defined(ATL_OS_SunOS)  /* retarded gcc on SunOS has no div */
Change line 58 of ATL_smm6x1x80_sse.c from:
#if MB <= 6
#if MB <= 6 || defined(ATL_OS_SunOS)  /* retarded gcc on SunOS has no div */

Search doesn't work with cpu throttling.

By default, newer Linuxes (and probably other OSes) have CPU throttling turned on even for desktops in order to save power. Since the speed of your CPU is constantly changing, the ATLAS timing results become essentially meaningless. Therefore, to get good performance numbers (and thus a fast ATLAS library), be sure to turn off cpu throttling in your BIOS before installation. If you are using the machine for high performance code a lot, you may want to leave it off. You can also usually turn off CPU throttling in your OS, see the ATLAS install guide (ATLAS/doc/atlas_install.pdf) for further details. Windows users may find these directions helpful.

ATLAS D and ZGEMM get wrong answer on MIPS for K=3

Save this file to ATLAS/tune/blas/gemm/CASES/ATL_dmm4x4xUR3_mips.c. If you have already installed, you may be able to fix your bad install by issuing these command in your BLDdir/tune/blas/gemm/ directory:
   make dinstall pre=d
   make zinstall pre=z
but the safest thing is to reinstall from source after saving the file.

Makefile error prevents TRSM tuning

Change both line 183 and 186 of ATLAS/makes/Make.l3tune from:
If you have already installed, you will need to reconfigure and reinstall in order to get the tuning.

ATLAS install fails on G4 when using gnu gcc instead of Apple's gcc

In addition to your normal configure flags, pass the following to configure when using gnu gcc on a system with AltiVec:
   -D c -DATL_AVgcc
We have reports that some users (depending on the way their gcc was installed) additionally had to pass the following to configure or configure fails to detect that their machine has the AltiVec unit, though we have not seen this problem in our own testing (again, probably due to the way we install gcc):

Makefile error can cause error when tuning matrix-vector multiply.

In ATLAS/makes/Make.mvtune we need to put quotes around every passing of the compiler macros [s,d,c,z]MVCC. Take this Make.mvtune and save it to your ATLAS/makes/Make.mvtune. If you have already configured, you must also copy it over your BLDdir/tune/blas/gemv/Makefile.

ATLAS configure fails to identify your G4 PowerBook under OS X

There is an error in ATLAS/CONFIG/src/backend/archinfo_freebsd.c. Change all mention of "c" to "k" on lines 81-87. Using vi you can accomplish this with the commands:
   vi ATLAS/CONFIG/src/backend/archinfo_freebsd.
   :81,87 s/"c/"k/g

atlas_prefetch.h won't compile using Sun CC v 5 or earlier.

In order to use prefetch with Sun's cc, ATLAS includes the Sun header file sun_prefetch.h, which did not exist until Sun CC version 6. ATLAS's architectural defaults use gcc anyway, which will almost certainly get you better performance. However, if you want to use Sun CC anyway, you will need to modify atlas_prefetch.h. I haven't had time to scope this problem myself (and I'm not sure I have access to a machine with an old enough Sun CC anyway), but here's a fix the user who originally found the error mentioned: Uncomment the version check on line 50 of atlas_prefetch.h Change line 50 of SRCdir/include/atlas_prefetch.h from:
#elif defined(__SUNPRO_C) && defined(__sparc) /* && __SUNPRO_CC > 0x600 */
#elif defined(__SUNPRO_C) && defined(__sparc) && __SUNPRO_CC >= 0x600
If you do this, you also want to install using gcc, and compare performance to see which is better.

Applying a patch to your ATLAS directory

To apply an ATLAS patch to your existing directory tree, save the patch file (we will call it patchfile from here on out) in your SRCdir/ directory, and then issue:
   patch -p1 < patchfile

Gcc compiler bug hurts PowerPC Performance.

ATLAS 3.8 assumes gcc 4.2 (most recent gcc stable release at ATLAS release). Gcc 4.2 is has a performance bug that normally cuts performance in half. ATLAS works around this by throwing the flags:
   -fno-schedule-insns -fno-rerun-loop-opt
This makes the performance drop caused by using gcc-4.2 roughly 1-7%, instead of 50%. If you are using a gcc version prior to gcc-4.2, remove the these flags from all your flag macros in your BLDdir/ after configure but before install to get back this performance. If you are using a later gcc release, you may be able to see if this bug has been fixed by scoping my original bug report .

Unkillable and relentless 'warning: the use of `tmpnam' is dangerous' warnings from gcc.

During configure you will get a lot of warnings of the following form:
/tmp/ccq5b8sE.o(.text+0x852): In function `CmndResults':
config.c: warning: the use of `tmpnam' is dangerous,
better use `mkstemp'

This is normal, and not an error. Let me translate this message out of gnu-speak:

  Hey, idiot, would you stop using that pesky ANSI/ISO C standard and
  use this non-standard routine instead?

For maximal compatability, ATLAS hews to the ANSI/ISO 9899-1990 standard, and so I cannot make the proposed change. Unfortunately, this warning message is literally immortal: there exists in gcc no flag combination that I can discover that can turn the freaking thing off. So, every time you link one of the config programs that calls this standard routine, the linker outputs this message, even if you turn on the strict ANSI compatibility flag. I reported it as an error to the gcc folks, but they point out it is the linker/glibc people that generate the "warning" and immediately closed the tracker. Still seems wrong to me that the strict ANSI flag with all warning messages turned off insists on printing out a message warning about standard usage, but there appears to be little for me to do about it. Therefore, just ignore the hectoring, and don't worry about these immortal, bogus, annoying and repetitive "warnings".

lapack library, which Apple does not currently provide, and so many users want to install the standard ATLAS. Unfortunately, when searching for libs the compiler looks in the system areas where apple keeps its ATLAS libs before looking in directories supplied by -L. This means that if you use -L and -l for your linking, you always get Apple's modified ATLAS, rather than the one you installed. There are two fixes for this problem that I know of. First, you can just link to the full name and path, rather than using -L. For instance, change something like:

   gcc -o xtst test.c -L /home/whaley/TEST/ATLAS/build64/lib -lcblas -latlas
   gcc -o xtst test.c /home/whaley/TEST/ATLAS/build64/lib/libcblas.a \
The only other trick I'm aware of is to rename your ATLAS libraries so that the Apple versions will not override them.

Basic compiler information

ATLAS has support for one or more compilers for every platform. In general, we provide gcc/gfortran for most supported architectures, since these compilers are freely available.

For each platform, ATLAS defaults to using the fastest available compiler. If the two compilers deliver roughly the same ATLAS performance, we pick gnu compilers, since they are freely available (assuming they compile on the platform in question).

The ATLAS architectural defaults were all build with gcc 4.2, except on IRIX/MIPS, where gcc was outperformed by SGI's cc. If you use gcc 4.0 or 4.1, then your performance will be cut roughly in half on all x86 platforms, so such users should install and use gcc 4.2. On the x86, Gcc 3 gets better performance than 4.1, but still less than 4.2. On gcc 3 platforms, we recommend you install gcc 4.2, and do a mixed gcc3/4 build, as described in the ATLAS install guide.

ATLAS is built to use gcc-4.2 as the main compiler. Newer versions are (hopefully) OK, but earlier ones are not. Be sure to run make time after completing your install to ensure that your compiler has not compromised your performance.

As of this writing, gcc-4.2 or newer (presumably) is necessary for all x86 archs. 4.1 or earlier will cut your performance by as much as half, depending on the architecture. The architectural defaults and compilers flags are unlikely to work for gcc-3.

For all PowerPC/POWER systems, gcc 4.2 is actually slower than 4.1, as described here. Gcc-3's late enough that they will take all the flags suggested by ATLAS seem to perform just fine.

What happens if I install with no Fortran compiler?

You enable this by passing --nof77 to configure. In this case, ATLAS will still install correctly, but it will obviously not create the Fortran77 interface libraries. You will not be able to run the testers under the BLDdir/interfaces/ directory, since these testers are written in Fortran. Further, ATLAS expects that you will be comparing against a Fortran77 interface BLAS, and this will obviously not be the case, and so you will need to make the following changes if you want to run any of the ATLAS tester/timers, even the ones written in C:

Installing with a non-default f77 compiler

The only Fortran routines in ATLAS are the Fortran77 interface routines, which do no computation. Therefore, the Fortran77 compiler has absolutely no effect on ATLAS's performance, and so the only reason you should need to use a non-default f77 compiler is if the f77 compiler you wish to use does not interoperate with ATLAS's default compiler.

To install with a non-default f77 compiler, simply override the default fortran compiler and flags from the command line when running config. This can be done by adding the following flags to configure:

   -C if /path/to/f77comp -F if 'f77 compiler flags'

Installing additional f77 interfaces

The only Fortran routines in ATLAS are the Fortran77 interface routines, which do no computation. Therefore, the Fortran77 compiler has absolutely no effect on ATLAS's performance, and so the only reason you should need to use a non-default f77 compiler is if the f77 compiler you wish to use does not interoperate with ATLAS's default compiler.

If you want to install ATLAS so it can be called from multiple, non-interoperable Fortran compilers (or indeed, have already installed with the wrong f77 compiler), you can do this with moderate ease, assuming you know how C and the given F77 compiler(s) interoperate. If you do not know this interoperational information, you must get configure to find it for you. To do this, create a bogus BLDdir directory (eg, mkdir bogus), and then run confgure from it, and overriding the default fortran compiler and flags from the command line as described here. You can then look at the generated's settings for the macro F2CDEFS and replicate them, along with the new F77 compiler/linker information, into your original

For those user's already aware of the information needed for C/F77 interoperation, ATLAS needs three pieces of information in order to correctly handle F77/C interoperation, and this information appears as defines to the C compiler, set in your's F2CDEFS.

The first macro controls the name space alterations necessary to make a C routine callable from Fortran77. The options are:

All F77-callable C routines should be lowercase, and have an underscore suffixed to their names.
All F77-callable C routines should be lowercase, have an underscore suffixed to their names, and if the F77 name itself posseses an underscore, two underscores should be suffixed.
All F77-callable C routines should be lowercase, with no name alteration.
All F77-callable C routines should be made uppercase, with no further name alteration.

The second macro provides a mapping between F77's INTEGER and the appropriate C integral type. Options are:

No definition
Default case where C's int corresponds to F77's INTEGER.
F77's INTEGER corresponds to C's long.
F77's INTEGER corresponds to C's short.

The third macro deals with F77 string handling. The options are:

The string's address is passed at the string's location on the stack, and the string's length is then passed as an F77_INTEGER after all explicit stack arguments.
Special option for CRAY machines, which uses Cray's fcd (fortran character descriptor) for interoperation.
The address of a structure is passed by a Fortran77 string, and the structure is of the form:
      struct {char *cp; F77_INTEGER len;};
A structure is passed by value for each Fortran77 string, and the structure is of the form:
      struct {char *cp; F77_INTEGER len;};

By default, ATLAS builds the F77 interface to the BLAS into the file pointed at by's F77BLASlib, and so changing this macro before recompiling the interface will allow you to build multiple F77 interfaces.

For example, say on a Solaris machine I want to build the f77 interface for both Sun's f77 and gfortran. First, I install ATLAS as normal, with the default gfortran compiler. Now, to get a f77 interface lib, I edit my ATLAS/Make.SunOS_SunUS2, and I find that ATLAS has detected the C/F77 interface for gfotran as: Sun's f77 compiler as:

   F2CDEFS = -DAdd__ -DStringSunStyle
I then change this to match f77:
   F2CDEFS = -DAdd_ -DStringSunStyle
Now, so that my gfortran interface will not be overwritten, I also change:
   F77BLASlib = $(LIBdir)/libf77blas.a
   F77BLASlib = $(LIBdir)/libsunf77blas.a
If I had built the threaded BLAS, I would make a similar change to PTF77BLASlib.

Finally, I change the f77 compiler/linker information from:

   F77 = /usr/local/bin/gfortran
   F77FLAGS = -O3 -funroll-all-loops
   F77 = /opt/SUNWspro/bin/f77
   F77FLAGS = -dalign -native -xarch=v8plusa -xO5
Now, I cd BLDdir/interfaces/blas/F77/src/, and issue:
   make clean
   make lib
If you are using threads, additionally issue:
   make ptlib
Now, when linking with Sun's f77, I link to -lsunf77blas.a -latlas.a, and when linking with g77 I use -lf77blas.a -latlas.a

You can essentially repeat this process for the LAPACK F77 interface, but change LAPACKlib rather than F77BLASlib, and go to BLDdir/interfaces/lapack/F77/src rather than BLDdir/interfaces/blas/F77/src. Also, LAPACK does not have a separate entry point for threads, so do not issue any of the additional threading instructions.

Finally, in your BLDdir/src/testing directory, issue :

   make clean ; make lib

How do I link with all these libraries?

The user libs created by ATLAS are:
The LAPACK routines provided by ATLAS. If you want a full lapack library, the .o in this lib can be archived into the f77 lapack lib without error.
The ANSI C interface to the BLAS.
The Fortran77 interface to the BLAS.
The ANSI C interface to the threaded (SMP) BLAS. This library only appears if you have asked for SMP support.
The Fortran77 interface to the threaded (SMP) BLAS. This library only appears if you have asked for SMP support.
The main ATLAS library, providing low-level routines for all interface libs.
If you have missing symbols on link, make sure you are linking in all of the libraries you need, and remember that order *is* significant. For instance, a code calling the Fortran77 interface to the BLAS would need:
   -L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -lf77blas -latlas
The full LAPACK library created by merging ATLAS and netlib LAPACK requires both C and Fortran77 interfaces, and thus that link line would be:
   -L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -llapack -lf77blas -lcblas -latlas
If you wish to use threaded BLAS, you simply indicate those interface libs rather than the sequential. The above line for SMP would be:
   -L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -llapack -lptf77blas -lptcblas -latlas

Why am I failing more LAPACK tester cases?

The LAPACK testers have been hand-tuned to work with the reference BLAS, and thus these BLAS almost always produce the least amount of failures (though you typically get some failures even with LAPACK's BLAS). If you scope the output files, you can quickly get an idea of which failures are serious by looking at the residuals. Residuals that are of size O(100) are typically not real failures, but merely a result of differing order of flops from what the ref blas do (which is legal and expected). In fact, on many platforms ATLAS achieves noticably less error than the reference BLAS, but since the tester has been so heavily tuned to the reference BLAS, these more accurate results cause more failures. You can see a simple example of this on most x86 platforms by compiling the reference BLAS with SSE only, which results in 64/32-bit precision. Then compile the same BLAS to use the x87 unit (which has 80-bit precision, though the gcc or the code can sometimes drop back to 64/32-bit precision briefly), and you will find you have more errors, despite having at least the same precision in all cases, and usually much more precision.

Real errors have residuals very large residuals (eg., 10e15). However, even these kinds of errors may be a result of the LAPACK tester being completely tuned to the LAPACK BLAS implementation. For instance, the DGER operation is supposed to do A += alpha * x * y. The reference BLAS perform A += x*(alpha*y). ATLAS does this or A += (alpha*x)*y, whichever is cheaper. This causes is the LIN testers to fail quite a few residual checks with size 10e15, even though both are legal. I have modified ATLAS to avoid these spurious GER problems, but not all of the LAPACK testers' reliance on fixed orderings can be fixed.

Installation help for AIX

Under AIX, you need to set the environment variable OBJECT_MODE appropriately. So, if you want a 64-bit library and you use the bash shell, you would do export OBJECT_MODE=64, while setenv OBJECT_MODE 32 would be for 32-bit libraries under csh.

Installing gcc under unix without being root

You do not need to be root to install a gcc that will deliver decent performance for ATLAS. I include below the exact steps I use to install the C compiler only in my own home area. Changing my home area path (given in the --prefix command to configure) to yours should allow you to do the same. These directions are for x86 users, where ATLAS needs gcc 4.2.x for decent performance. They work pretty much the same for other platforms/gcc versions. Note that these directions will install gfortran as well. The fortran compiler is not needed for ATLAS performance, so if you want to use a different fortran compiler than this version of gfortran, simply omit fortran from the --enable-languages step. NOTE: if your old gcc version is a lot older than the new one, there will often by library incompatabilities, which can cause linking, particularly of FORTRAN codes, to fail. If this occurs, you can either use the old compiler to do linking, or set the environment variable LD_LIBRARY_PATH so that your new libraries are seen before the system libraries. For instance, on the above install where the system libraries were for gcc 4.1.0, I had to add the following line to my .cshrc:
   setenv LD_LIBRARY_PATH /home/whaley/local/gcc-4.2.1/lib:/home/whaley/local/gcc-4.2.1/lib64

Post install tuning.

Here are some tips to improving ATLAS performance after an install:

Improving ATLAS small case performance by changing malloc behavior

ATLAS allocates a buffer space for most GEMM calls. When I wrote it, my assumption was that only first call requires a switch to kernel space to do the allocation, and incurs the unneeded overhead of zeroing out the memory. However, by default Linux (as well as some other OSes, such as OS X) allocates non-trivial sized allocations using mmap, which means that when free is called, the memory is immediately returned to the system. Thus all malloc calls have extremely high overheads.

This is not a big problem if you are doing a large matrix multiply, where the cubic computation disguises this square cost. For small problems, though, the O(N**2) costs are actually dominant, and this type of malloc behavior effectively doubles them (at least). You should be able to change Linux's malloc behavior by setting these environment variables:

   setenv MALLOC_MMAP_MAX_ 0

Once this is done, malloc should be cheaper, but ATLAS was tuned with the expensive malloc. Therefore, you may be able to get better small-case performance by rerunning the crossover search with these environment variables set (don't do this unless you are going to keep these settings whenever you use this library). You can rerun the search from the ATLAS/tune/blas/gemm/ARCH directory by issuing:

   make sRun_tfc pre=s
   make dRun_tfc pre=d
   make cRun_tfc pre=c
   make zRun_tfc pre=z
This search takes a *loooong* time, then to build the changes into the libraries, go to ATLAS/bin/ARCH, and issue:
   make xsl3blastst
   make xdl3blastst
   make xcl3blastst
   make xzl3blastst

Tuning CacheEdge.

CacheEdge is an Level 2 Cache blocking parameter; because it's effects are fairly subtle on most machines, it often goes wrong on machines experiencing any kind of load, causing performance to be be suboptimal. CacheEdge can improve performance by as much as 15%, and it can reduce ATLAS's memory usage as well.

In ATLAS/tune/blas/gemm/ARCH, issue make xdfindCE. Run

   ./xdfindCE -m [N] -n [N] -k [N]
where [N} is replaced by a very large number that is a multiple of your blocking factor. You want to make this number as large as you can stand to wait on, and this varies a great deal from machine to machine. A good guestimate for most machines might be around 2000.

You want to run this program several times to get a consensus idea of what a good setting would be. If a CacheEdge setting gets performance in the same range as no CacheEdge (CacheEdge of 0 is no CacheEdge in printout of xdfindCE), it is still recommended that you use that setting, since ATLAS with CacheEdge set will use less memory as problem sizes grows.

Once you have gotten an idea of what to set CacheEdge to, you can change it by editing ATLAS/include/ARCH/atlas_cacheedge.h. xdfindCE prints out data in KB, but atlas_cacheedge.h needs bytes, so multiply the xdfindCE result by 1024 to get the number you want to use in atlas_cacheedge.h.

Let's take an example. Say xdfindCE printed out this:

TA  TB       M       N       K   alpha    beta  CacheEdge       TIME    MFLOPS
==  ==  ======  ======  ======  ======  ======  =========  =========  ========

 T   N    1000    1000    1000    1.00    1.00          0      5.470    365.63
 T   N    1000    1000    1000    1.00    1.00         16      5.470    365.63
 T   N    1000    1000    1000    1.00    1.00         32      5.460    366.30
 T   N    1000    1000    1000    1.00    1.00         64      5.470    365.63
 T   N    1000    1000    1000    1.00    1.00        128      5.260    380.23
 T   N    1000    1000    1000    1.00    1.00        256      5.240    381.68

Initial CE=256KB, mflop=381.68

Best CE=256KB, mflop=381.68
So we want to set CacheEdge to 1024*256 = 262144. atlas_cacheedge will look something like:
   #define CacheEdge 196608
If your initial install did not use CacheEdge, line 3 will be missing completely. If you don't have this line, you would simply add it, using the new value of 262144. In the above example, we would simply replace 196608 with 262144.

By successively editing this file and recompiling, for instance ATLAS/bin/ARCH/x[d,s,z,c]mmtst you can tune this value further. Many users expect that they should set CacheEdge to the actual size of their L2 cache. This is only rarely the best setting, mainly because L2 caches are normally combined data/instruction, and so a smaller setting, leaving room for instruction caching, is usually best. On some machines with large L2 caches, things like associativity, or even TLB issues, can make it more efficient to use a very small subset of the available cache.

Once you have set CacheEdge to the value you need, update all libs with the new setting by issuing make xdl3blastst xsl3blastst xcl3blastst xzl3blastst in your ATLAS/bin/ARCH directory.

Special hints for setting CacheEdge for multiprocessor machines

CacheEdge turns out to be very important to threaded performance. Unfortunately most of the default CacheEdge settings were obtained on single processor machines. So, you may well be able to see a substantial speedup by changing CacheEdge for your multiprocessor system.

The basic technique for finding CacheEdge is given here. Unfortunately, xdfindCE presently operates only on uniprocessor code, so what you want to use instead is varying CacheEdge and iteratively compiling and running x[pre]l3blastst_pt until you have a number you are happy with. It is vital to use a large problem. Use the largest problem you can stand to wait on for this many timing runs.

x[pre]findCE usually takes the smallest CacheEdge setting possible, since this saves memory. For multiprocessor systems, however, it is vital to use as much of the available cache as possible so that the processors spend as little time contending for the bus as possible. Thus, you want to set CacheEdge to the largest value that gives decent results. I usually run xdfindCE a few times to get an idea of ranges, and then try the larger settings by running x[pre]l3blastst_pt. Remember that threaded timings have to use walltime, so make sure any speedup is repeatable before changing CacheEdge.

Building a complete LAPACK library

ATLAS does not provide a full LAPACK library. However, there is a simple way to get ATLAS to provide its faster LAPACK routines to a full LAPACK library provided by netlib. First, install lapack from netlib. First, download and install the standard LAPACK library from the LAPACK homepage. Then when doing the configure step to ATLAS pass the following flag:

If you need to add full LAPACK after ATLAS has already been built, in your BLDdir/lib/ directory (where you should have a liblapack.a), issue the following commands:

  mkdir tmp
  cd tmp
  ar x ../liblapack.a
  cp <your LAPACK path & lib> ../liblapack.a
  ar r ../liblapack.a *.o
  cd ..
  rm -rf tmp

Just linking in ATLAS's liblapack.a first will not get you the best LAPACK performance, mainly because LAPACK's untuned ILAENV will be used instead of ATLAS's tuned one. So, if you use any LAPACK routine that is not provided by ATLAS, it is essential that you create this hybrid LAPACK/ATLAS library in order to get the best performance.

My performance drops off for very large problem (N > 1500)

Note this is for the serial interface. If it is only your parallel performance that drops off for large problem sizes, you are probably experiencing this problem.

A serial performance cliff is usually due to the normal install failing to set CacheEdge to any value, and then eventually ATLAS winds up using memory-saving algorithms that hurt performance. The solution is to set CacheEdge, so we use less workspace, while improving overall performance.

What you want to do is tune CachEdge, as shown here, but be sure to use very large problem sizes in order find CacheEdge.

Your install dies with "unable to get timings in tolerance"

This means that ATLAS could not get repeatable timings. There are several things that could cause this to happen. This could occur if the machine is heavily loaded or experiences a sudden surge in usage from another program, for instance. If this is the problem, simply keep restarting the install (as discussed below) until it finishes.

Another problem that could cause this is that ATLAS misdetected the peak of your machine, and is thus using an inadequate timing interval. You can see if this is happening by scoping how long each timing is taking. If it is very quick, and thus unrepeatable, you need to tell ATLAS to pump up the timing granularity. To do this, edit the files BLDdir/include/atlas_?sysinfo.h. Each of these four files will have a quantity called ATL_nkflop. Pump this quantity up by some significant factor until timings are regular. I usually increase it by a factor of 5 or 10. If the individual timings are then too slow, interrupt the process, and decrease the values.

Finally, the Level 1 timings very often display this problem even when the timing interval is sufficient. The most likely explaination of this non-repeatable timing problem involves inadequate cache flushing, but it has not been tracked down for sure. Regardless, the only way is to keep restarting the interrupted install until it completes, as explained below.

Most of the time, when an install dies in this way, you can just restart it, as outlined here. If this dies right away in the exact same timing, but without actually running the timing again, it means that the install process kept a record of the bad timing, and is just rereading it. You then need to remove the bogus timing record file. This file will be in the appropriate BLDdir directory under the res/ subdirectory. For instance, if you are dying in the level 1 tuning, the result files are stored in BLDdir/tune/blas/level1/res, and if you are in the gemm tuning they are in BLDdir/tune/blas/gemm/ARCH/res, etc. Your last message from the dying install should give you most info you need to figure out the result directory and the file. Just remove the file (or all the files of that precision, if you cannot figure out the specific file that is bad), and restart the install.

How do I restart an interrupted install?

If your ATLAS install was interrupted, and you have fixed the problem, you can usually safely (there are always exceptions; if the install died in the middle of an ar command, for instance, many systems cannot recover) restart the install by:

How do I do I get rid of all the .o's?

Once you have done make install (and/or manually copied the libraries and include files you want), you can simply delete the entire OBJdir directory.

Help for building ATLAS under windows

You must install ATLAS in an directory without spaces in the path. Eg., /home/Administrator/ATLAS is fine, but /cygwin/c/My Home Area is not.

Building ATLAS with a non-cygwin compiler

If you want to build ATLAS with a non-cygwin compiler (i.e., a native windows compiler such as Intel's icl or ifort), you will need to perform the following steps (you can skip this if using the gnu compilers is OK):
  1. Set your environment variables correctly
  2. Tell config what compilers and flags to use
  3. Do some post-build fiddling
You will also need to install as root, as ATLAS builds unix-workalike compiler wrappers and puts them in /usr/local/bin.

Setting your Windows compiler environment correctly

The easiest way to get your environment variables setup is to modify your /cygdrive/c/cygwin/cygwin.bat file. In order to be able to call MSVC++ (cl) from cygwin, I added the following lines to my cygwin.bat:
chdir c:\Program Files\Microsoft Visual Studio 8\VC
call vcvarsall
Obviously, adapt the path as necessary to point to your MSVC++ install. You can use the 'find files' function to search for the vcvarsall file.

For the Intel compilers, the C compiler (icl) seemed to work out-of-the-box once I had done the MSVC++ environment variables setup outlined above. To get the Fortran compiler (ifort) rolling, I had to additinally add:

chdir c:\Program Files\Intel\Compiler\Fortran\10.0.025\IA32\Bin
call ifortvars
to my cygwin.bat file. In all, here is the /cygdrive/c/cygwin/cygwin.bat that allows me to use MSVC++, icl, and ifort:
@echo off
chdir c:\Program Files\Microsoft Visual Studio 8\VC
call vcvarsall
chdir c:\Program Files\Intel\Compiler\Fortran\10.0.025\IA32\Bin
call ifortvars
chdir C:\cygwin\bin
bash --login -i

These same changes can be made to /usr/X11R6/bin/starxwin.bat if you use that batchfile to start xterms instead of cygwin windows.

If you need to set up the environment after bash has already started running, things are a little more difficult. Take this file, modify the paths to match your system, and then source it from the bash prompt.
Back to Window compiler overview

Telling config about your windows compilers

Presently, ATLAS allows you to change only the interfaces compilers with the native Windows compilers (the bulk of the library must be compiled with gcc).

In order to use ifort for the F77 interfaces, add the flag

   -C if ifort
to your configure command.

In order to compile the C interface routines with CL (the command line interface to MSVC++), add the flag:

   -C ic cl

It may be important to have a library that you can link without linking to libcygwin.a, which is a GPL (not LGPL) library. To say that you want to avoid this library, pass the following flag to configure:

   -Si nocygwin 1
Note that without the cygwin library, you cannot build the threaded BLAS, since Windows does not provide POSIX threads compliance, so only throw this flag if you need it.

Back to Window compiler overview

Post-config fiddling for windows compilers

If you are compiling with all gnu compilers, no changes need to be made.

Some of the time, depending on how your programming environment is setup, you also don't need to change anything to use ifort as your linker. Occasionally, in order to link using the using ifort, with the rest of the library (including the interface routines) compiled with gcc, you need to explicitly link to both gcc's libgcc.a (which uses the LGPL, not GPL, license), and MSVC's libcmt.lib. Unfortunately, the spaces in the typical windows paths pose a challenge when called from the commandline, and so you typically need to copy the file over to the cygwin directory (logical links don't work). Therefore, before modifying or, issue a command like (adjust the paths to your system):

   cp /cygdrive/c/Program\ Files/Microsoft\ Visual\ Studio\ 8/VC/lib/libcmt.lib /usr/local/lib/.

Wait until the ATLAS build is complete before modifying the to point at these libraries (but you must make these changes before doing make check or make time). First, try to do a make check before trying these mods, as you may not need to make them! If this fails, set your LIBS flag in something like (modifying the paths as necessary):

  LIBS = /cygdrive/c/cygwin/usr/local/lib/libcmt.lib \
If you built without cygwin, then the line you want is:
   LIBS = /cygdrive/c/cygwin/usr/local/lib/libcmt.lib \
and then the ATLAS timers and testers should work. You will probably need to link to these libraries yourself when linking to ATLAS externally as well.

Back to Window compiler overview

Make shared fails with something like: ld: cannot find -lgfortran

This is due to configure choosing a bad F77SYSLIB (in due to a faulty probe. This macro should include the path where your can be found. You can fix this problem by manually changing this macro to have the correct path after configure. If you have this problem, simply fix this macro, and then reissue your make shared commands.

What you should set it to will vary by system, as you need to locate the correct library path. You can usually do this using the locate command. For instance, here's what I see on my system:

My ATLAS install is 64-bit, so I would set (in
   F77SYSLIB = -L/usr/lib/gcc/x86_64-linux-gnu/4.2/ -lgfortran
If you have multiple choices and don't know how to choose, it should work to try them until your shared build works.

Problems with gcc 4.2 on Windows

A user has reported that this gcc bug can cause problems when ATLAS is installed on Windows using gcc 4.2. The fix given in the gnu bug report (passing -fno-common) has been reported to fix the ATLAS install as well. Therefore, if you are using gcc 4 under windows, you may want to add the following to your configure command:
   -Fa alg -fno-common

Gcc's violation of the x86 ABI causes seg faults/bus errors when mixed with other compilers

Gcc violates the x86-32 ABI by mandating a 16-byte aligned stack, where the ABI mandates a 4-byte aligned stack. Therefore, if gcc is used to compile some routines that may be called from an ABI-compliant compiler, you may (depending on how lucky you are) get a seg fault or a bus error. Therefore, if you plan on mixing gcc with any compiler that does not extend the ABI in the same way, or if you want to use windows threads, you must tell gcc to turn off this misfeature, which may be done by passing -mpreferred-stack-boundary=2 to all gnu compilers. This problem has been noticed on Windows, but might occur anywhere gcc is not used to generate all of the object files. If you are compiling ATLAS with all gnu compilers, you can just pass:
   -Fa alg -mpreferred-stack-boundary=2
to configure. If you are building ATLAS with some mix of non-gnu and gnu compilers, then you need the above flag added only to the gnu compilers. You can do this at configure time by setting it for individual compilers, as described here, or you can configure as normal, but edit before starting the build, and add the flag to all gnu compilers' flags.

Note that if you add -mpreferred-stack-boundary=2 to the fortran compiler flags, then g77/gfortran will cause the BLAS tester to die with a bus error/seg fault, because gfortran then aligns double precision arrays to 4-byte rather than 8-byte boundaries. ATLAS assumes native alignment on data types, and so the code seg faults. As long as you don't need to call the Fortran interface from a non-gnu compiler, then you can solve this problem by simply leaving the -mpreferred-stack-boundary flag off of for the Fortran interface. At configure time, you would pass:

   -Fa acg -mpreferred-stack-boundary=2

I reported these related problems (ABI non-conformance and non-native alignment) to the gcc folks, and got a range of opinions, the most startling of which was that whatever they do is the standard. On that link, one of the more helpful replies mentions that the non-native alignment problem should be fixed in gcc 4.4, but that the ABI non-compliance must be maintained in order to match the prior failure to comply with the ABI.

When linking ATLAS's testers, I'm getting a bunch of undefined BLAS symbols (eg. dgemm_, dgemv_, etc).

The ATLAS BLAS testers (x[s,d,c,z]l[1,2,3]blastst) expect to compare against a F77 interface BLAS library for performance and testing purposes. You get these missing symbols when your Make.ARCH's BLASlib is left blank, or does not point at a complete BLAS library. If you have a non-ATLAS BLAS built somewhere, point the BLASlib macro at it. If you don't, probably the easiest fix is probably to grab the Fortran77 reference BLAS tarfile, and build it into the required lib. If you don't want to do this, or don't have access to Fortran77, then you can have ATLAS test against its own C reference as discussed here.

I'm linking with C, and getting missing symbols (such as w_wsfe, do_fio, w_esfe or s_stop).

These kinds of symbols are Fortran library calls. The problem is that the C linker does not automatically find the Fortran libraries. The most common fix is to either link using your fortran linker, or to rewrite your code so that Fortran routines are not called. If you know where they are, you can also choose to link in the Fortran libraries explicitly.

Problems with linking/missing LAPACK routines on OS X

OS X has a built-in version of ATLAS, and uses the standard names for them. They may be less up-to-date and/or have less libs than something you install yourself; in particular, if you have a Fortran compiler, you can build a full lapack library, which Apple does not currently provide, and so many users want to install the standard ATLAS. Unfortunately, when searching for libs the compiler looks in the system areas where apple keeps its ATLAS libs before looking in directories supplied by -L. This means that if you use -L and -l for your linking, you always get Apple's modified ATLAS, rather than the one you installed. There are two fixes for this problem that I know of. First, you can just link to the full name and path, rather than using -L. For instance, change something like:
   gcc -o xtst test.c -L /home/whaley/TEST/ATLAS/build64/lib -lcblas -latlas
   gcc -o xtst test.c /home/whaley/TEST/ATLAS/build64/lib/libcblas.a \
The only other trick I'm aware of is to rename your ATLAS libraries so that the Apple versions will not override them.

How about C++ header files for the C interfaces?

Since ATLAS does not provide full OO C++ interfaces, I am reluctant to raise the expectation that it does by providing C++ specific header files. What I have always envisioned is the C++ programmer creating his own include files, such as:
>cat cppblas.h
extern "C" {
#include cblas.h

>cat cpplapack.h
extern "C" {
#include clapack.h
If you are a C++ programmer using ATLAS, and think differently, let me know.

How do I restart an install from scratch?

Simply do a rm -rf * in your BLDdir directory, and then reconfigure and build.

My system doesn't have the -f option to cp

If you take the following line, and put it in a file cp you make executable, and then put it in your path before your system cp, it should get rid of the -f option:
/bin/cp `echo $* | sed -e 's/-f / /'`

[Home] [Docs] [FAQ] [Errata] [Software] [Install] [Support] [Lists] [Developer home] [Timings]