ATLAS 3.8.1 errata


[Home] [Docs] [FAQ] [Errata] [Software] [Install] [Support] [Lists] [Developer home] [Timings]

ATLAS v3.8.1 released 02/22/08

ATLAS errors:

Known issues:

Noteworthy compiler issues:

System problems & user tips:


[Home] [Docs] [FAQ] [Errata] [Software] [Support] [Lists] [Developer home] [Timings]

SourceForge Logo ICL Logo

Bad GEMM call causes performance drop for all architectures

A piece of duplicated code causes a special-case code to be used for all GEMM problems, which reduces performance on all architectures and almost all problems. To fix, comment out (or delete) lines 191-194 of ATLAS/src/blas/gemm/ATL_gemmXX.c, which read:
         {
            mm2 = mm1;
            mm1 = Mjoin(PATL,mmJITcp);
         }
(notice these 4 lines are a duplicate of the for lines above.

Complex GEMM sometimes accesses C when BETA=0

This happens when K is much larger than M and N, and is caused by a bug in special-case GEMM code. To fix, edit ATLAS/src/blas/gemm/ATL_cmmJITcp.c, and change lines 267 and 268 from:
   else  /* two or more dim < NB, requires generated cleanup */
      NBmm0 = NBmm1 = NBmmX = Mjoin(PATLU,pKBmm);
to:
   else { NBmm0 = NBmm1 = NBmmX = Mjoin(PATLU,pKBmm);
          if (SCALAR_IS_ZERO(beta)) Mjoin(PATL,gezero)(M, N, C, ldc); }

ATLAS threaded performance dies for large problems

On some platforms, if you time very large problems, you'll see that ATLAS's threaded library does well, and then suddenly drops to below serial performance. This is probably an error in how we recover from lack of memory in the threaded case, but in the meantime the easy fix is to increase the total amount of memory ATLAS is allowed to allocate. To do this, edit the file ATLAS/include/atlas_lvl3.h, and pump up the macro ATL_MaxMalloc, which is the maximal size (in bytes) ATLAS is allowed to allocate. It is presently set to be 64MB; make it as large as you think you can afford.

Search doesn't work with cpu throttling.

By default, newer Linuxes (and probably other OSes) have CPU throttling turned on even for desktops in order to save power. Since the speed of your CPU is constantly changing, the ATLAS timing results become essentially meaningless. Therefore, to get good performance numbers (and thus a fast ATLAS library), be sure to turn off cpu throttling in your BIOS before installation. If you are using the machine for high performance code a lot, you may want to leave it off. You can also usually turn off CPU throttling in your OS, see the ATLAS install guide (ATLAS/doc/atlas_install.pdf) for further details. Windows users may find these directions helpful.

atlas_prefetch.h won't compile using Sun CC v 5 or earlier.

In order to use prefetch with Sun's cc, ATLAS includes the Sun header file sun_prefetch.h, which did not exist until Sun CC version 6. ATLAS's architectural defaults use gcc anyway, which will almost certainly get you better performance. However, if you want to use Sun CC anyway, you will need to modify atlas_prefetch.h. I haven't had time to scope this problem myself (and I'm not sure I have access to a machine with an old enough Sun CC anyway), but here's a fix the user who originally found the error mentioned: Uncomment the version check on line 50 of atlas_prefetch.h Change line 50 of SRCdir/include/atlas_prefetch.h from:
#elif defined(__SUNPRO_C) && defined(__sparc) /* && __SUNPRO_CC > 0x600 */
to:
#elif defined(__SUNPRO_C) && defined(__sparc) && __SUNPRO_CC >= 0x600
If you do this, you also want to install using gcc, and compare performance to see which is better.

Applying a patch to your ATLAS directory

To apply an ATLAS patch to your existing directory tree, save the patch file (we will call it patchfile from here on out) in your SRCdir/ directory, and then issue:
   patch -p1 < patchfile

Gcc compiler bug hurts PowerPC Performance.

ATLAS 3.8 assumes gcc 4.2 (most recent gcc stable release at ATLAS release). Gcc 4.2 is has a performance bug that normally cuts performance in half. ATLAS works around this by throwing the flags:
   -fno-schedule-insns -fno-rerun-loop-opt
This makes the performance drop caused by using gcc-4.2 roughly 1-7%, instead of 50%. If you are using a gcc version prior to gcc-4.2, remove the these flags from all your flag macros in your BLDdir/Make.inc after configure but before install to get back this performance. If you are using a later gcc release, you may be able to see if this bug has been fixed by scoping my original bug report .

Unkillable and relentless 'warning: the use of `tmpnam' is dangerous' warnings from gcc.

During configure you will get a lot of warnings of the following form:
/tmp/ccq5b8sE.o(.text+0x852): In function `CmndResults':
config.c: warning: the use of `tmpnam' is dangerous,
better use `mkstemp'

This is normal, and not an error. Let me translate this message out of gnu-speak:

  Hey, idiot, would you stop using that pesky ANSI/ISO C standard and
  use this non-standard routine instead?

For maximal compatability, ATLAS hews to the ANSI/ISO 9899-1990 standard, and so I cannot make the proposed change. Unfortunately, this warning message is literally immortal: there exists in gcc no flag combination that I can discover that can turn the freaking thing off. So, every time you link one of the config programs that calls this standard routine, the linker outputs this message, even if you turn on the strict ANSI compatibility flag. I reported it as an error to the gcc folks, but they point out it is the linker/glibc people that generate the "warning" and immediately closed the tracker. Still seems wrong to me that the strict ANSI flag with all warning messages turned off insists on printing out a message warning about standard usage, but there appears to be little for me to do about it. Therefore, just ignore the hectoring, and don't worry about these immortal, bogus, annoying and repetitive "warnings".

Testing ATLAS with the LAPACK testers.

It takes a bit of hoop-jumping to get ATLAS to pass the LAPACK testers. First, the LAPACK testers have an error in them that causes them to flag the ATLAS TRSM as bad. This is explained briefly in the LAPACK errata. You can either ignore the incorrect errors generated by lapack in the xlintst? testers, or you can override ATLAS's TRSM in order to make them go away. To override, edit LAPACK's make.inc, and link in the F77 [c,s,d,z]trsm.o in BLASLIB before the ATLAS libs.

After this, you'll still get failures reported (in our last run on Linux_PIIISSE1, we had failures in cnep.out, csep.out, and ced.out), but these failures will happen with stock LAPACK and the F77BLAS.

Your install dies with "unable to get timings in tolerance"

This means that ATLAS could not get repeatable timings. There are several things that could cause this to happen. This could occur if the machine is heavily loaded or experiences a sudden surge in usage from another program, for instance. If this is the problem, simply keep restarting the install (as discussed below) until it finishes.

Another problem that could cause this is that ATLAS misdetected the peak of your machine, and is thus using an inadequate timing interval. You can see if this is happening by scoping how long each timing is taking. If it is very quick, and thus unrepeatable, you need to tell ATLAS to pump up the timing granularity. To do this, edit the files BLDdir/include/atlas_?sysinfo.h. Each of these four files will have a quantity called ATL_nkflop. Pump this quantity up by some significant factor until timings are regular. I usually increase it by a factor of 5 or 10. If the individual timings are then too slow, interrupt the process, and decrease the values.

Finally, the Level 1 timings very often display this problem even when the timing interval is sufficient. The most likely explaination of this non-repeatable timing problem involves inadequate cache flushing, but it has not been tracked down for sure. Regardless, the only way is to keep restarting the interrupted install until it completes, as explained below.

Most of the time, when an install dies in this way, you can just restart it, as outlined here. If this dies right away in the exact same timing, but without actually running the timing again, it means that the install process kept a record of the bad timing, and is just rereading it. You then need to remove the bogus timing record file. This file will be in the appropriate BLDdir directory under the res/ subdirectory. For instance, if you are dying in the level 1 tuning, the result files are stored in BLDdir/tune/blas/level1/res, and if you are in the gemm tuning they are in BLDdir/tune/blas/gemm/ARCH/res, etc. Your last message from the dying install should give you most info you need to figure out the result directory and the file. Just remove the file (or all the files of that precision, if you cannot figure out the specific file that is bad), and restart the install.

Installing with a non-default f77 compiler

The only Fortran routines in ATLAS are the Fortran77 interface routines, which do no computation. Therefore, the Fortran77 compiler has absolutely no effect on ATLAS's performance, and so the only reason you should need to use a non-default f77 compiler is if the f77 compiler you wish to use does not interoperate with ATLAS's default compiler.

To install with a non-default f77 compiler, simply override the default fortran compiler and flags from the command line when running config. This can be done by adding the following flags to configure:

   -C if /path/to/f77comp -F if 'f77 compiler flags'

Installing additional f77 interfaces

The only Fortran routines in ATLAS are the Fortran77 interface routines, which do no computation. Therefore, the Fortran77 compiler has absolutely no effect on ATLAS's performance, and so the only reason you should need to use a non-default f77 compiler is if the f77 compiler you wish to use does not interoperate with ATLAS's default compiler.

If you want to install ATLAS so it can be called from multiple, non-interoperable Fortran compilers (or indeed, have already installed with the wrong f77 compiler), you can do this with moderate ease, assuming you know how C and the given F77 compiler(s) interoperate. If you do not know this interoperational information, you must get configure to find it for you. To do this, create a bogus BLDdir directory (eg, mkdir bogus), and then run confgure from it, and overriding the default fortran compiler and flags from the command line as described here. You can then look at the generated Make.inc's settings for the macro F2CDEFS and replicate them, along with the new F77 compiler/linker information, into your original Make.inc.

For those user's already aware of the information needed for C/F77 interoperation, ATLAS needs three pieces of information in order to correctly handle F77/C interoperation, and this information appears as defines to the C compiler, set in your Make.inc's F2CDEFS.

The first macro controls the name space alterations necessary to make a C routine callable from Fortran77. The options are:

Add_
All F77-callable C routines should be lowercase, and have an underscore suffixed to their names.
Add__
All F77-callable C routines should be lowercase, have an underscore suffixed to their names, and if the F77 name itself posseses an underscore, two underscores should be suffixed.
NoChange
All F77-callable C routines should be lowercase, with no name alteration.
UpCase
All F77-callable C routines should be made uppercase, with no further name alteration.

The second macro provides a mapping between F77's INTEGER and the appropriate C integral type. Options are:

No definition
Default case where C's int corresponds to F77's INTEGER.
F77_INTEGER=long
F77's INTEGER corresponds to C's long.
F77_INTEGER=short
F77's INTEGER corresponds to C's short.

The third macro deals with F77 string handling. The options are:

StringSunStyle
The string's address is passed at the string's location on the stack, and the string's length is then passed as an F77_INTEGER after all explicit stack arguments.
CrayStyle
Special option for CRAY machines, which uses Cray's fcd (fortran character descriptor) for interoperation.
StringStructPtr
The address of a structure is passed by a Fortran77 string, and the structure is of the form:
      struct {char *cp; F77_INTEGER len;};
StringStructVal
A structure is passed by value for each Fortran77 string, and the structure is of the form:
      struct {char *cp; F77_INTEGER len;};

By default, ATLAS builds the F77 interface to the BLAS into the file pointed at by Make.inc's F77BLASlib, and so changing this macro before recompiling the interface will allow you to build multiple F77 interfaces.

For example, say on a Solaris machine I want to build the f77 interface for both Sun's f77 and gfortran. First, I install ATLAS as normal, with the default gfortran compiler. Now, to get a f77 interface lib, I edit my ATLAS/Make.SunOS_SunUS2, and I find that ATLAS has detected the C/F77 interface for gfotran as: Sun's f77 compiler as:

   F2CDEFS = -DAdd__ -DStringSunStyle
I then change this to match f77:
   F2CDEFS = -DAdd_ -DStringSunStyle
Now, so that my gfortran interface will not be overwritten, I also change:
   F77BLASlib = $(LIBdir)/libf77blas.a
to:
   F77BLASlib = $(LIBdir)/libsunf77blas.a
If I had built the threaded BLAS, I would make a similar change to PTF77BLASlib.

Finally, I change the f77 compiler/linker information from:

   F77 = /usr/local/bin/gfortran
   F77FLAGS = -O3 -funroll-all-loops
to:
   F77 = /opt/SUNWspro/bin/f77
   F77FLAGS = -dalign -native -xarch=v8plusa -xO5
Now, I cd BLDdir/interfaces/blas/F77/src/, and issue:
   make clean
   make lib
If you are using threads, additionally issue:
   make ptlib
Now, when linking with Sun's f77, I link to -lsunf77blas.a -latlas.a, and when linking with g77 I use -lf77blas.a -latlas.a

You can essentially repeat this process for the LAPACK F77 interface, but change LAPACKlib rather than F77BLASlib, and go to BLDdir/interfaces/lapack/F77/src rather than BLDdir/interfaces/blas/F77/src. Also, LAPACK does not have a separate entry point for threads, so do not issue any of the additional threading instructions.

Finally, in your BLDdir/src/testing directory, issue :

   make clean ; make lib

How do I link with all these libraries?

The user libs created by ATLAS are:
liblapack.a
The LAPACK routines provided by ATLAS. If you want a full lapack library, the .o in this lib can be archived into the f77 lapack lib without error.
libcblas.a
The ANSI C interface to the BLAS.
libf77blas.a
The Fortran77 interface to the BLAS.
libptcblas.a
The ANSI C interface to the threaded (SMP) BLAS. This library only appears if you have asked for SMP support.
libptf77blas.a
The Fortran77 interface to the threaded (SMP) BLAS. This library only appears if you have asked for SMP support.
libatlas.a
The main ATLAS library, providing low-level routines for all interface libs.
If you have missing symbols on link, make sure you are linking in all of the libraries you need, and remember that order *is* significant. For instance, a code calling the Fortran77 interface to the BLAS would need:
   -L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -lf77blas -latlas
The full LAPACK library created by merging ATLAS and netlib LAPACK requires both C and Fortran77 interfaces, and thus that link line would be:
   -L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -llapack -lf77blas -lcblas -latlas
If you wish to use threaded BLAS, you simply indicate those interface libs rather than the sequential. The above line for SMP would be:
   -L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -llapack -lptf77blas -lptcblas -latlas

Basic compiler information

ATLAS has support for one or more compilers for every platform. In general, we provide gcc/gfortran for most supported architectures, since these compilers are freely available.

For each platform, ATLAS defaults to using the fastest available compiler. If the two compilers deliver roughly the same ATLAS performance, we pick gnu compilers, since they are freely available (assuming they compile on the platform in question).

The ATLAS architectural defaults were all build with gcc 4.2, except on IRIX/MIPS, where gcc was outperformed by SGI's cc. If you use gcc 4.0 or 4.1, then your performance will be cut roughly in half on all x86 platforms, so such users should install and use gcc 4.2. On the x86, Gcc 3 gets better performance than 4.1, but still less than 4.2. On gcc 3 platforms, we recommend you install gcc 4.2, and do a mixed gcc3/4 build, as described in the ATLAS install guide.

ATLAS is built to use gcc-4.2 as the main compiler. Newer versions are (hopefully) OK, but earlier ones are not. Be sure to run make time after completing your install to ensure that your compiler has not compromised your performance.

As of this writing, gcc-4.2 or newer (presumably) is necessary for all x86 archs. 4.1 or earlier will cut your performance by as much as half, depending on the architecture. The architectural defaults and compilers flags are unlikely to work for gcc-3.

For all PowerPC/POWER systems, gcc 4.2 is actually slower than 4.1, as described here. Gcc-3's late enough that they will take all the flags suggested by ATLAS seem to perform just fine.

My system doesn't have the -f option to cp

If you take the following line, and put it in a file cp you make executable, and then put it in your path before your system cp, it should get rid of the -f option:
/bin/cp `echo $* | sed -e 's/-f / /'`

Building a complete LAPACK library

ATLAS does not provide a full LAPACK library. However, there is a simple way to get ATLAS to provide its faster LAPACK routines to a full LAPACK library provided by netlib. First, install lapack from netlib. First, download and install the standard LAPACK library from the LAPACK homepage. Then when doing the configure step to ATLAS pass the following flag:
   --with-netlib-lapack=[path/to/lapack/lib]

If you need to add full LAPACK after ATLAS has already been built, in your BLDdir/lib/ directory (where you should have a liblapack.a), issue the following commands:

  mkdir tmp
  cd tmp
  ar x ../liblapack.a
  cp <your LAPACK path & lib> ../liblapack.a
  ar r ../liblapack.a *.o
  cd ..
  rm -rf tmp

Just linking in ATLAS's liblapack.a first will not get you the best LAPACK performance, mainly because LAPACK's untuned ILAENV will be used instead of ATLAS's tuned one. So, if you use any LAPACK routine that is not provided by ATLAS, it is essential that you create this hybrid LAPACK/ATLAS library in order to get the best performance.

How do I restart an install from scratch?

Simply do a rm -rf * in your BLDdir directory, and then reconfigure and build.

How do I restart an interrupted install?

If your ATLAS install was interrupted, and you have fixed the problem, you can usually safely (there are always exceptions; if the install died in the middle of an ar command, for instance, many systems cannot recover) restart the install by:

How do I do I get rid of all the .o's?

Once you have done make install (and/or manually copied the libraries and include files you want), you can simply delete the entire OBJdir directory.

Do NOT use the -fno-f2c flag with g77

Haven't tracked this down in a while, but it appears to break quite a few things in fairly non-obvious ways for mixed g77/gcc libs.

What happens if I install with no Fortran compiler?

You enable this by passing --nof77 to configure. In this case, ATLAS will still install correctly, but it will obviously not create the Fortran77 interface libraries. You will not be able to run the testers under the BLDdir/interfaces/ directory, since these testers are written in Fortran. Further, ATLAS expects that you will be comparing against a Fortran77 interface BLAS, and this will obviously not be the case, and so you will need to make the following changes if you want to run any of the ATLAS tester/timers, even the ones written in C:

My performance drops off for very large problem (N > 1500)

Note this is for the serial interface. If it is only your parallel performance that drops off for large problem sizes, you are probably experiencing this problem.

A serial performance cliff is usually due to the normal install failing to set CacheEdge to any value, and then eventually ATLAS winds up using memory-saving algorithms that hurt performance. The solution is to set CacheEdge, so we use less workspace, while improving overall performance.

What you want to do is tune tune CachEdge, as shown here, but be sure to use very large problem sizes in order find CacheEdge.

Post install tuning.

Here are some tips to improving ATLAS performance after an install:

Tuning CacheEdge.

CacheEdge is an Level 2 Cache blocking parameter; because it's effects are fairly subtle on most machines, it often goes wrong on machines experiencing any kind of load, causing performance to be be suboptimal. CacheEdge can improve performance by as much as 15%, and it can reduce ATLAS's memory usage as well.

In ATLAS/tune/blas/gemm/ARCH, issue make xdfindCE. Run

   ./xdfindCE -m [N] -n [N] -k [N]
where [N} is replaced by a very large number that is a multiple of your blocking factor. You want to make this number as large as you can stand to wait on, and this varies a great deal from machine to machine. A good guestimate for most machines might be around 2000.

You want to run this program several times to get a consensus idea of what a good setting would be. If a CacheEdge setting gets performance in the same range as no CacheEdge (CacheEdge of 0 is no CacheEdge in printout of xdfindCE), it is still recommended that you use that setting, since ATLAS with CacheEdge set will use less memory as problem sizes grows.

Once you have gotten an idea of what to set CacheEdge to, you can change it by editing ATLAS/include/ARCH/atlas_cacheedge.h. xdfindCE prints out data in KB, but atlas_cacheedge.h needs bytes, so multiply the xdfindCE result by 1024 to get the number you want to use in atlas_cacheedge.h.

Let's take an example. Say xdfindCE printed out this:

TA  TB       M       N       K   alpha    beta  CacheEdge       TIME    MFLOPS
==  ==  ======  ======  ======  ======  ======  =========  =========  ========

 T   N    1000    1000    1000    1.00    1.00          0      5.470    365.63
 T   N    1000    1000    1000    1.00    1.00         16      5.470    365.63
 T   N    1000    1000    1000    1.00    1.00         32      5.460    366.30
 T   N    1000    1000    1000    1.00    1.00         64      5.470    365.63
 T   N    1000    1000    1000    1.00    1.00        128      5.260    380.23
 T   N    1000    1000    1000    1.00    1.00        256      5.240    381.68

Initial CE=256KB, mflop=381.68


Best CE=256KB, mflop=381.68
So we want to set CacheEdge to 1024*256 = 262144. atlas_cacheedge will look something like:
#ifndef ATLAS_CACHEEDGE_H
   #define ATLAS_CACHEEDGE_H
   #define CacheEdge 196608
#endif
If your initial install did not use CacheEdge, line 3 will be missing completely. If you don't have this line, you would simply add it, using the new value of 262144. In the above example, we would simply replace 196608 with 262144.

By successively editing this file and recompiling, for instance ATLAS/bin/ARCH/x[d,s,z,c]mmtst you can tune this value further. Many users expect that they should set CacheEdge to the actual size of their L2 cache. This is only rarely the best setting, mainly because L2 caches are normally combined data/instruction, and so a smaller setting, leaving room for instruction caching, is usually best. On some machines with large L2 caches, things like associativity, or even TLB issues, can make it more efficient to use a very small subset of the available cache.

Once you have set CacheEdge to the value you need, update all libs with the new setting by issuing make xdl3blastst xsl3blastst xcl3blastst xzl3blastst in your ATLAS/bin/ARCH directory.

Special hints for setting CacheEdge for multiprocessor machines

CacheEdge turns out to be very important to threaded performance. Unfortunately most of the default CacheEdge settings were obtained on single processor machines. So, you may well be able to see a substantial speedup by changing CacheEdge for your multiprocessor system.

The basic technique for finding CacheEdge is given here. Unfortunately, xdfindCE presently operates only on uniprocessor code, so what you want to use instead is varying CacheEdge and iteratively compiling and running x[pre]l3blastst_pt until you have a number you are happy with. It is vital to use a large problem. Use the largest problem you can stand to wait on for this many timing runs.

x[pre]findCE usually takes the smallest CacheEdge setting possible, since this saves memory. For multiprocessor systems, however, it is vital to use as much of the available cache as possible so that the processors spend as little time contending for the bus as possible. Thus, you want to set CacheEdge to the largest value that gives decent results. I usually run xdfindCE a few times to get an idea of ranges, and then try the larger settings by running x[pre]l3blastst_pt. Remember that threaded timings have to use walltime, so make sure any speedup is repeatable before changing CacheEdge.

Changing ATLAS's maximum buffer space.

Another way to tune ATLAS to your system is to vary the amount of buffer space ATLAS is allowed to allocate. In general, you want to set this as high as you can without causing swapping. If you have a machine with low memory, and you see dramatic slowdowns as the problem size goes up, you should definitely choose a smaller max size. ATLAS defaults to 4MB (except on the x86-64, where we default to 8MB). To vary this value, edit the file:
   ATLAS/include/atlas_lvl3.h
And set ATL_MaxMalloc to the maximal number of bytes that you want ATLAS to allocate internally. After editing, rebuild the libs by issuing:
   make xdl3blastst xcl3blastst xzl3blastst xsl3blastst
in your ATLAS/bin/ARCH directory.

Improving ATLAS small case performance by changing malloc behavior

ATLAS allocates a buffer space for most GEMM calls. When I wrote it, my assumption was that only first call requires a switch to kernel space to do the allocation, and incurs the unneeded overhead of zeroing out the memory. However, by default Linux (as well as some other OSes, such as OS X) allocates non-trivial sized allocations using mmap, which means that when free is called, the memory is immediately returned to the system. Thus all malloc calls have extremely high overheads.

This is not a big problem if you are doing a large matrix multiply, where the cubic computation disguises this square cost. For small problems, though, the O(N**2) costs are actually dominant, and this type of malloc behavior effectively doubles them (at least). You should be able to change Linux's malloc behavior by setting these environment variables:

   setenv MALLOC_TRIM_THRESHOLD_ -1
   setenv MALLOC_MMAP_MAX_ 0

Once this is done, malloc should be cheaper, but ATLAS was tuned with the expensive malloc. Therefore, you may be able to get better small-case performance by rerunning the crossover search with these environment variables set (don't do this unless you are going to keep these settings whenever you use this library). You can rerun the search from the ATLAS/tune/blas/gemm/ARCH directory by issuing:

   make sRun_tfc pre=s
   make dRun_tfc pre=d
   make cRun_tfc pre=c
   make zRun_tfc pre=z
This search takes a *loooong* time, then to build the changes into the libraries, go to ATLAS/bin/ARCH, and issue:
   make xsl3blastst
   make xdl3blastst
   make xcl3blastst
   make xzl3blastst

When linking ATLAS's testers, I'm getting a bunch of undefined BLAS symbols (eg. dgemm_, dgemv_, etc).

The ATLAS BLAS testers (x[s,d,c,z]l[1,2,3]blastst) expect to compare against a F77 interface BLAS library for performance and testing purposes. You get these missing symbols when your Make.ARCH's BLASlib is left blank, or does not point at a complete BLAS library. If you have a non-ATLAS BLAS built somewhere, point the BLASlib macro at it. If you don't, probably the easiest fix is probably to grab the Fortran77 reference BLAS tarfile, and build it into the required lib. If you don't want to do this, or don't have access to Fortran77, then you can have ATLAS test against its own C reference as discussed here.

I'm linking with C, and getting missing symbols (such as w_wsfe, do_fio, w_esfe or s_stop).

These kinds of symbols are Fortran library calls. The problem is that the C linker does not automatically find the Fortran libraries. The most common fix is to either link using your fortran linker, or to rewrite your code so that Fortran routines are not called. If you know where they are, you can also choose to link in the Fortran libraries explicitly.

Installing gcc under unix without being root

You do not need to be root to install a gcc that will deliver decent performance for ATLAS. I include below the exact steps I use to install the C compiler only in my own home area. Changing my home area path (given in the --prefix command to configure) to yours should allow you to do the same. These directions are for x86 users, where ATLAS needs gcc 4.2.x for decent performance. They work pretty much the same for other platforms/gcc versions. Note that these directions will install gfortran as well. The fortran compiler is not needed for ATLAS performance, so if you want to use a different fortran compiler than this version of gfortran, simply omit fortran from the --enable-languages step. NOTE: if your old gcc version is a lot older than the new one, there will often by library incompatabilities, which can cause linking, particularly of FORTRAN codes, to fail. If this occurs, you can either use the old compiler to do linking, or set the environment variable LD_LIBRARY_PATH so that your new libraries are seen before the system libraries. For instance, on the above install where the system libraries were for gcc 4.1.0, I had to add the following line to my .cshrc:
   setenv LD_LIBRARY_PATH /home/whaley/local/gcc-4.2.1/lib:/home/whaley/local/gcc-4.2.1/lib64

Help for building ATLAS under windows

You must install ATLAS in an directory without spaces in the path. Eg., /home/Administrator/ATLAS is fine, but /cygwin/c/My Home Area is not.

By default, ATLAS will use a very low-res CPU timer. If you are installing on a fairly unloaded machine, you can improve your install enormously by using ATLAS's cycle-accurate walltimer. If you are using gcc as your C interface compiler, enable the cycle-accurate walltimer by passing these flags to configure:

   -D c -DPentiumCPS=[your Mhz]
Obviously, [Your Mhz] should be replaced by the speed in MegaHertz of your machine (eg. I passed -D c -DPentiumCPS=2200 to configure on my 2.2Ghz Athlon-64).

If your C interface compiler is MSVC++ or Intel's C compiler, you can still improve the accuracy of timings on an unloaded machine by passing the flags:

   -D c -DWALL

Building ATLAS with a non-cygwin compiler

If you want to build ATLAS with a non-cygwin compiler (i.e., a native windows compiler such as Intel's icl or ifort), you will need to perform the following steps (you can skip this if using the gnu compilers is OK):
  1. Set your environment variables correctly
  2. Tell config what compilers and flags to use
  3. Do some post-build Make.inc fiddling
You will also need to install as root, as ATLAS builds unix-workalike compiler wrappers and puts them in /usr/local/bin.

Setting your Windows compiler environment correctly

The easiest way to get your environment variables setup is to modify your /cygdrive/c/cygwin/cygwin.bat file. In order to be able to call MSVC++ (cl) from cygwin, I added the following lines to my cygwin.bat:
chdir c:\Program Files\Microsoft Visual Studio 8\VC
call vcvarsall
Obviously, adapt the path as necessary to point to your MSVC++ install. You can use the 'find files' function to search for the vcvarsall file.

For the Intel compilers, the C compiler (icl) seemed to work out-of-the-box once I had done the MSVC++ environment variables setup outlined above. To get the Fortran compiler (ifort) rolling, I had to additinally add:

chdir c:\Program Files\Intel\Compiler\Fortran\10.0.025\IA32\Bin
call ifortvars
to my cygwin.bat file. In all, here is the /cygdrive/c/cygwin/cygwin.bat that allows me to use MSVC++, icl, and ifort:
@echo off
chdir c:\Program Files\Microsoft Visual Studio 8\VC
call vcvarsall
chdir c:\Program Files\Intel\Compiler\Fortran\10.0.025\IA32\Bin
call ifortvars
C:
chdir C:\cygwin\bin
bash --login -i

These same changes can be made to /usr/X11R6/bin/starxwin.bat if you use that batchfile to start xterms instead of cygwin windows.

If you need to set up the environment after bash has already started running, things are a little more difficult. Take this file, modify the paths to match your system, and then source it from the bash prompt.
Back to Window compiler overview

Telling config about your windows compilers

Presently, ATLAS allows you to change only the interfaces compilers with the native Windows compilers (the bulk of the library must be compiled with gcc).

In order to use ifort for the F77 interfaces, add the flag

   -C if ifort
to your configure command.

In order to compile the C interface routines with CL (the command line interface to MSVC++), add the flag:

   -C ic cl

It may be important to have a library that you can link without linking to libcygwin.a, which is a GPL (not LGPL) library. To say that you want to avoid this library, pass the following flag to configure:

   -Si nocygwin 1
Note that without the cygwin library, you cannot build the threaded BLAS, since Windows does not provide POSIX threads compliance, so only throw this flag if you need it.

Back to Window compiler overview

Post-config Make.inc fiddling for windows compilers

If you are compiling with all gnu compilers, no changes need to be made.

Some of the time, depending on how your programming environment is setup, you also don't need to change anything to use ifort as your linker. Occasionally, in order to link using the using ifort, with the rest of the library (including the interface routines) compiled with gcc, you need to explicitly link to both gcc's libgcc.a (which uses the LGPL, not GPL, license), and MSVC's libcmt.lib. Unfortunately, the spaces in the typical windows paths pose a challenge when called from the commandline, and so you typically need to copy the file over to the cygwin directory (logical links don't work). Therefore, before modifying or Make.inc, issue a command like (adjust the paths to your system):

   cp /cygdrive/c/Program\ Files/Microsoft\ Visual\ Studio\ 8/VC/lib/libcmt.lib /usr/local/lib/.

Wait until the ATLAS build is complete before modifying the Make.inc to point at these libraries (but you must make these changes before doing make check or make time). First, try to do a make check before trying these mods, as you may not need to make them! If this fails, set your LIBS flag in Make.inc something like (modifying the paths as necessary):

  LIBS = /cygdrive/c/cygwin/usr/local/lib/libcmt.lib \
         /cygdrive/c/cygwin/lib/libcygwin.a
If you built without cygwin, then the line you want is:
   LIBS = /cygdrive/c/cygwin/usr/local/lib/libcmt.lib \
          /cygdrive/c/cygwin/lib/gcc/i686-pc-mingw32/3.4.4/libgcc.a 
and then the ATLAS timers and testers should work. You will probably need to link to these libraries yourself when linking to ATLAS externally as well.

Back to Window compiler overview

Using pthreads under Windows

Right now, the only way to use pthreads under Windows is if you compile with gcc. Due to the aforementioned gcc bug, this currently means that your Fortran compiler must be g77. If this is acceptable, no extra steps are needed: just run config as normal, and say you want to use threads.

The reason ATLAS presently requires gcc to get you threaded support is that the supported Windows compilers don't supply the required include file pthread.h. My MSVC++ is version 6, and I suspect newer versions actually do provide pthreads. If they do, you can try to install, and it may work. If it does, however, you will still have a small problem. I am aware of no wall-timer under Windows, and you need a wall-timer to get meaningful timings of threaded code. ATLAS defaults to calling a cpu timer, which will make the install work, but will not give you any idea how fast your lib is. Therefore, you will need to turn on ATLAS's assembler timer.

To do this, before you do 'make install', Change line 110 of ATLAS/makes/Make.sysinfo from:

        $(CC) -c $(CCFLAGS) ATL_walltime.c
to:
        $(BC) -c $(BCFLAGS) ATL_walltime.c

Now, before install, but after config, edit the Make.ARCH created by config, and add the definition -DPentiumCPS=[MyMhz] to the CDEFS macro. For instance, on my 2.5Ghz P4, I would change the line:

   CDEFS = $(L2SIZE) $(INCLUDES) $(F2CDEFS) $(ARCHDEFS)
to:
   CDEFS = $(L2SIZE) $(INCLUDES) $(F2CDEFS) $(ARCHDEFS) -DPentiumCPS=2500

How about C++ header files for the C interfaces?

Since ATLAS does not provide full OO C++ interfaces, I am reluctant to raise the expectation that it does by providing C++ specific header files. What I have always envisioned is the C++ programmer creating his own include files, such as:
>cat cppblas.h
extern "C" {
#include cblas.h
}

>cat cpplapack.h
extern "C" {
#include clapack.h
}
If you are a C++ programmer using ATLAS, and think differently, let me know.

Problems with linking/missing LAPACK routines on OS X

OS X has a built-in version of ATLAS, and uses the standard names for them. They may be less up-to-date and/or have less libs than something you install yourself; in particular, if you have a Fortran compiler, you can build a full lapack library, which Apple does not currently provide, and so many users want to install the standard ATLAS. Unfortunately, when searching for libs the compiler looks in the system areas where apple keeps its ATLAS libs before looking in directories supplied by -L. This means that if you use -L and -l for your linking, you always get Apple's modified ATLAS, rather than the one you installed. There are two fixes for this problem that I know of. First, you can just link to the full name and path, rather than using -L. For instance, change something like:
   gcc -o xtst test.c -L /home/whaley/TEST/ATLAS/build64/lib -lcblas -latlas
to:
   gcc -o xtst test.c /home/whaley/TEST/ATLAS/build64/lib/libcblas.a \
          /home/whaley/TEST/ATLAS/build64/lib/libatlas.a
The only other trick I'm aware of is to rename your ATLAS libraries so that the Apple versions will not override them.

Why am I failing more LAPACK tester cases?

The LAPACK testers have been hand-tuned to work with the reference BLAS, and thus these BLAS almost always produce the least amount of failures (though you typically get some failures even with LAPACK's BLAS). If you scope the output files, you can quickly get an idea of which failures are serious by looking at the residuals. Residuals that are of size O(100) are typically not real failures, but merely a result of differing order of flops from what the ref blas do (which is legal and expected). In fact, on many platforms ATLAS achieves noticably less error than the reference BLAS, but since the tester has been so heavily tuned to the reference BLAS, these more accurate results cause more failures. You can see a simple example of this on most x86 platforms by compiling the reference BLAS with SSE only, which results in 64/32-bit precision. Then compile the same BLAS to use the x87 unit (which has 80-bit precision, though the gcc or the code can sometimes drop back to 64/32-bit precision briefly), and you will find you have more errors, despite having at least the same precision in all cases, and usually much more precision.

Real errors have residuals very large residuals (eg., 10e15). However, even these kinds of errors may be a result of the LAPACK tester being completely tuned to the LAPACK BLAS implementation. For instance, the DGER operation is supposed to do A += alpha * x * y. The reference BLAS perform A += x*(alpha*y). ATLAS does this or A += (alpha*x)*y, whichever is cheaper. This causes is the LIN testers to fail quite a few residual checks with size 10e15, even though both are legal. I have modified ATLAS to avoid these spurious GER problems, but not all of the LAPACK testers' reliance on fixed orderings can be fixed.

Installation help for AIX

Under AIX, you need to set the environment variable OBJECT_MODE appropriately. So, if you want a 64-bit library and you use the bash shell, you would do export OBJECT_MODE=64, while setenv OBJECT_MODE 32 would be for 32-bit libraries under csh.
[Home] [Docs] [FAQ] [Errata] [Software] [Install] [Support] [Lists] [Developer home] [Timings]