{
mm2 = mm1;
mm1 = Mjoin(PATL,mmJITcp);
}
(notice these 4 lines are a duplicate of the for lines above.
Complex GEMM sometimes accesses C when BETA=0
This happens when K is much larger than M and N, and is caused by a bug
in special-case GEMM code. To fix, edit
ATLAS/src/blas/gemm/ATL_cmmJITcp.c, and change lines 267 and 268 from:
else /* two or more dim < NB, requires generated cleanup */
NBmm0 = NBmm1 = NBmmX = Mjoin(PATLU,pKBmm);
to:
else { NBmm0 = NBmm1 = NBmmX = Mjoin(PATLU,pKBmm);
if (SCALAR_IS_ZERO(beta)) Mjoin(PATL,gezero)(M, N, C, ldc); }
Search doesn't work with cpu throttling.
By default, newer Linuxes (and probably other OSes) have CPU throttling
turned on even for desktops
in order to save power. Since the speed of your CPU is constantly changing,
the ATLAS timing results become essentially meaningless. Therefore, to get
good performance numbers (and thus a fast ATLAS library), be sure to turn
off cpu throttling in your BIOS before installation. If you are using
the machine for high performance code a lot, you may want to leave it off.
You can also usually turn off CPU throttling in your OS, see
the ATLAS install guide (ATLAS/doc/atlas_install.pdf) for
further details. Windows users may find
these directions helpful.
atlas_prefetch.h won't compile using Sun CC
v 5 or earlier.
In order to use prefetch with Sun's cc, ATLAS includes the Sun header file
sun_prefetch.h, which did not exist until Sun CC version 6.
ATLAS's architectural defaults use gcc anyway, which will almost certainly
get you better performance.
However, if you want to use Sun CC anyway, you will need
to modify atlas_prefetch.h. I haven't had time to scope this problem
myself (and I'm not sure I have access to a machine with an old enough
Sun CC anyway), but here's a fix the user who originally found the error
mentioned: Uncomment the version check on line 50 of atlas_prefetch.h
Change line 50 of SRCdir/include/atlas_prefetch.h from:
#elif defined(__SUNPRO_C) && defined(__sparc) /* && __SUNPRO_CC > 0x600 */to:
#elif defined(__SUNPRO_C) && defined(__sparc) && __SUNPRO_CC >= 0x600If you do this, you also want to install using gcc, and compare performance to see which is better.
Applying a patch to your ATLAS directory
To apply an ATLAS patch to your existing directory tree, save the patch
file (we will call it patchfile from here on out) in your
SRCdir/ directory, and then issue:
patch -p1 < patchfile
Gcc compiler bug hurts PowerPC Performance.
ATLAS 3.8 assumes gcc 4.2 (most recent gcc stable release at ATLAS release).
Gcc 4.2 is has a performance bug that normally cuts performance in half.
ATLAS works around this by throwing the flags:
-fno-schedule-insns -fno-rerun-loop-opt
This makes the performance drop caused by using gcc-4.2 roughly 1-7%, instead
of 50%. If you are using a gcc version prior to gcc-4.2, remove the these
flags from all your flag macros in your BLDdir/Make.inc after configure
but before install to get back this performance. If you are using a later
gcc release, you may be able to see if this bug has been fixed by scoping my
original bug report
.
Unkillable and relentless
'warning: the use of `tmpnam' is dangerous' warnings from gcc.
During configure you will get a lot of warnings of the following form:
/tmp/ccq5b8sE.o(.text+0x852): In function `CmndResults': config.c: warning: the use of `tmpnam' is dangerous, better use `mkstemp'
This is normal, and not an error. Let me translate this message out of gnu-speak:
Hey, idiot, would you stop using that pesky ANSI/ISO C standard and use this non-standard routine instead?
For maximal compatability, ATLAS hews to the ANSI/ISO 9899-1990 standard, and so I cannot make the proposed change. Unfortunately, this warning message is literally immortal: there exists in gcc no flag combination that I can discover that can turn the freaking thing off. So, every time you link one of the config programs that calls this standard routine, the linker outputs this message, even if you turn on the strict ANSI compatibility flag. I reported it as an error to the gcc folks, but they point out it is the linker/glibc people that generate the "warning" and immediately closed the tracker. Still seems wrong to me that the strict ANSI flag with all warning messages turned off insists on printing out a message warning about standard usage, but there appears to be little for me to do about it. Therefore, just ignore the hectoring, and don't worry about these immortal, bogus, annoying and repetitive "warnings".
Testing ATLAS with the LAPACK testers.
It takes a bit of hoop-jumping to get ATLAS to pass the LAPACK testers.
First, the LAPACK testers have an error in them that causes them to flag the
ATLAS TRSM as bad. This is explained briefly in the
LAPACK errata.
You can either ignore the incorrect errors generated by lapack in the
xlintst? testers, or you can override ATLAS's TRSM in order to make them
go away. To override, edit LAPACK's make.inc, and link in the
F77 [c,s,d,z]trsm.o in BLASLIB before the ATLAS libs.
After this, you'll still get failures reported (in our last run on Linux_PIIISSE1, we had failures in cnep.out, csep.out, and ced.out), but these failures will happen with stock LAPACK and the F77BLAS.
Your install dies with "unable to get timings in tolerance"
This means that ATLAS could not get repeatable timings. There are several
things that could cause this to happen. This could occur if the machine is
heavily loaded or experiences a sudden surge in usage from another program,
for instance. If this is the problem, simply keep restarting the install
(as discussed below) until it finishes.
Another problem that could cause this is that ATLAS misdetected the peak of your machine, and is thus using an inadequate timing interval. You can see if this is happening by scoping how long each timing is taking. If it is very quick, and thus unrepeatable, you need to tell ATLAS to pump up the timing granularity. To do this, edit the files BLDdir/include/atlas_?sysinfo.h. Each of these four files will have a quantity called ATL_nkflop. Pump this quantity up by some significant factor until timings are regular. I usually increase it by a factor of 5 or 10. If the individual timings are then too slow, interrupt the process, and decrease the values.
Finally, the Level 1 timings very often display this problem even when the timing interval is sufficient. The most likely explaination of this non-repeatable timing problem involves inadequate cache flushing, but it has not been tracked down for sure. Regardless, the only way is to keep restarting the interrupted install until it completes, as explained below.
Most of the time, when an install dies in this way, you can just restart it, as outlined here. If this dies right away in the exact same timing, but without actually running the timing again, it means that the install process kept a record of the bad timing, and is just rereading it. You then need to remove the bogus timing record file. This file will be in the appropriate BLDdir directory under the res/ subdirectory. For instance, if you are dying in the level 1 tuning, the result files are stored in BLDdir/tune/blas/level1/res, and if you are in the gemm tuning they are in BLDdir/tune/blas/gemm/ARCH/res, etc. Your last message from the dying install should give you most info you need to figure out the result directory and the file. Just remove the file (or all the files of that precision, if you cannot figure out the specific file that is bad), and restart the install.
Installing with a non-default f77 compiler
The only Fortran routines in ATLAS are the Fortran77 interface routines,
which do no computation. Therefore, the Fortran77 compiler has absolutely
no effect on ATLAS's performance, and so the only reason you should need
to use a non-default f77 compiler is if the f77 compiler you wish to use
does not interoperate with ATLAS's default compiler.
To install with a non-default f77 compiler, simply override the default fortran compiler and flags from the command line when running config. This can be done by adding the following flags to configure:
-C if /path/to/f77comp -F if 'f77 compiler flags'
Installing additional f77 interfaces
The only Fortran routines in ATLAS are the Fortran77 interface routines,
which do no computation. Therefore, the Fortran77 compiler has absolutely
no effect on ATLAS's performance, and so the only reason you should need
to use a non-default f77 compiler is if the f77 compiler you wish to use
does not interoperate with ATLAS's default compiler.
If you want to install ATLAS so it can be called from multiple, non-interoperable Fortran compilers (or indeed, have already installed with the wrong f77 compiler), you can do this with moderate ease, assuming you know how C and the given F77 compiler(s) interoperate. If you do not know this interoperational information, you must get configure to find it for you. To do this, create a bogus BLDdir directory (eg, mkdir bogus), and then run confgure from it, and overriding the default fortran compiler and flags from the command line as described here. You can then look at the generated Make.inc's settings for the macro F2CDEFS and replicate them, along with the new F77 compiler/linker information, into your original Make.inc.
For those user's already aware of the information needed for C/F77 interoperation, ATLAS needs three pieces of information in order to correctly handle F77/C interoperation, and this information appears as defines to the C compiler, set in your Make.inc's F2CDEFS.
The first macro controls the name space alterations necessary to make a C routine callable from Fortran77. The options are:
The second macro provides a mapping between F77's INTEGER and the appropriate C integral type. Options are:
The third macro deals with F77 string handling. The options are:
struct {char *cp; F77_INTEGER len;};
struct {char *cp; F77_INTEGER len;};
By default, ATLAS builds the F77 interface to the BLAS into the file pointed at by Make.inc's F77BLASlib, and so changing this macro before recompiling the interface will allow you to build multiple F77 interfaces.
For example, say on a Solaris machine I want to build the f77 interface for both Sun's f77 and gfortran. First, I install ATLAS as normal, with the default gfortran compiler. Now, to get a f77 interface lib, I edit my ATLAS/Make.SunOS_SunUS2, and I find that ATLAS has detected the C/F77 interface for gfotran as: Sun's f77 compiler as:
F2CDEFS = -DAdd__ -DStringSunStyleI then change this to match f77:
F2CDEFS = -DAdd_ -DStringSunStyleNow, so that my gfortran interface will not be overwritten, I also change:
F77BLASlib = $(LIBdir)/libf77blas.ato:
F77BLASlib = $(LIBdir)/libsunf77blas.aIf I had built the threaded BLAS, I would make a similar change to PTF77BLASlib.
Finally, I change the f77 compiler/linker information from:
F77 = /usr/local/bin/gfortran F77FLAGS = -O3 -funroll-all-loopsto:
F77 = /opt/SUNWspro/bin/f77 F77FLAGS = -dalign -native -xarch=v8plusa -xO5Now, I cd BLDdir/interfaces/blas/F77/src/, and issue:
make clean make libIf you are using threads, additionally issue:
make ptlibNow, when linking with Sun's f77, I link to -lsunf77blas.a -latlas.a, and when linking with g77 I use -lf77blas.a -latlas.a
You can essentially repeat this process for the LAPACK F77 interface, but change LAPACKlib rather than F77BLASlib, and go to BLDdir/interfaces/lapack/F77/src rather than BLDdir/interfaces/blas/F77/src. Also, LAPACK does not have a separate entry point for threads, so do not issue any of the additional threading instructions.
Finally, in your BLDdir/src/testing directory, issue :
make clean ; make lib
How do I link with all these libraries?
The user libs created by ATLAS are:
-L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -lf77blas -latlasThe full LAPACK library created by merging ATLAS and netlib LAPACK requires both C and Fortran77 interfaces, and thus that link line would be:
-L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -llapack -lf77blas -lcblas -latlasIf you wish to use threaded BLAS, you simply indicate those interface libs rather than the sequential. The above line for SMP would be:
-L$(MY_HOME)/ATLAS/lib/$(MY_ARCH)/ -llapack -lptf77blas -lptcblas -latlas
For each platform, ATLAS defaults to using the fastest available compiler.
If the two compilers deliver roughly the same ATLAS performance, we pick
gnu compilers, since they are freely available (assuming they compile
on the platform in question).
The ATLAS architectural defaults were all build with gcc 4.2, except on
IRIX/MIPS, where gcc was outperformed by SGI's cc. If you use
gcc 4.0 or 4.1, then your performance will be cut roughly in half on
all x86 platforms, so
such users should install and use gcc 4.2. On the x86, Gcc 3 gets better
performance than 4.1, but still less than 4.2. On gcc 3 platforms,
we recommend you install gcc 4.2, and do a mixed gcc3/4 build, as described
in the
ATLAS install guide.
ATLAS is built to use gcc-4.2 as the main compiler. Newer versions are
(hopefully) OK, but earlier ones are not. Be sure to run
make time after completing your install to ensure that your compiler
has not compromised your performance.
As of this writing, gcc-4.2 or newer (presumably) is necessary for all x86
archs. 4.1 or earlier
will cut your performance by as much as half, depending on the architecture.
The architectural defaults and compilers flags are unlikely to work for
gcc-3.
For all PowerPC/POWER systems, gcc 4.2 is actually slower than 4.1, as described
here. Gcc-3's late enough that they will take all
the flags suggested by ATLAS seem to perform just fine.
If you need to add full LAPACK after ATLAS has already been built,
in your BLDdir/lib/ directory (where you should have a
liblapack.a), issue the following commands:
Just linking in ATLAS's liblapack.a first will not get you the best LAPACK
performance, mainly because LAPACK's untuned ILAENV will be used instead
of ATLAS's tuned one. So, if you use any LAPACK routine that is not
provided by ATLAS, it is essential that you create this hybrid LAPACK/ATLAS
library in order to get the best performance.
A serial performance cliff is
usually due to the normal install failing to set CacheEdge to any
value, and then eventually ATLAS winds up using memory-saving algorithms
that hurt performance. The solution is to set CacheEdge, so we use less
workspace, while improving overall performance.
What you want to do is tune tune CachEdge, as shown here,
but be sure to use very large problem sizes in order find CacheEdge.
In ATLAS/tune/blas/gemm/ARCH, issue make xdfindCE. Run
You want to run
this program several times to get a consensus idea of what a good setting
would be. If a CacheEdge setting gets performance in the same range as
no CacheEdge (CacheEdge of 0 is no CacheEdge in printout of xdfindCE),
it is still recommended that you use that setting, since ATLAS with
CacheEdge set will use less memory as problem sizes grows.
Once you have gotten an idea of what to set CacheEdge to, you can change it by
editing ATLAS/include/ARCH/atlas_cacheedge.h. xdfindCE
prints out data in KB, but atlas_cacheedge.h needs bytes, so multiply
the xdfindCE result by 1024 to get the number you want to use in
atlas_cacheedge.h.
Let's take an example. Say xdfindCE printed out this:
By successively editing this file and recompiling, for instance
ATLAS/bin/ARCH/x[d,s,z,c]mmtst you can tune this value further.
Many users expect that they should set CacheEdge to the actual size of their
L2 cache. This is only rarely the best setting, mainly because L2 caches
are normally combined data/instruction, and so a smaller setting,
leaving room for instruction caching, is usually best. On some machines
with large L2 caches, things like associativity, or even TLB issues, can
make it more efficient to use a very small subset of the available cache.
Once you have set CacheEdge to the value you need, update all libs
with the new setting by issuing
make xdl3blastst xsl3blastst xcl3blastst xzl3blastst in your
ATLAS/bin/ARCH directory.
The basic technique for finding CacheEdge is given
here. Unfortunately, xdfindCE
presently operates only on uniprocessor code, so what you want
to use instead is varying CacheEdge and iteratively compiling
and running x[pre]l3blastst_pt until you have a number
you are happy with. It is vital to use a large problem. Use
the largest problem you can stand to wait on for this many timing runs.
x[pre]findCE usually takes the smallest CacheEdge setting
possible, since this saves memory. For multiprocessor systems, however,
it is vital to use as much of the available cache as possible so that
the processors spend as little time contending for the bus as possible.
Thus, you want to set CacheEdge to the largest value that gives
decent results. I usually run xdfindCE a few times to get an
idea of ranges, and then try the larger settings by running
x[pre]l3blastst_pt. Remember that threaded timings have
to use walltime, so make sure any speedup is repeatable before changing
CacheEdge.
This is not a big problem if you are doing a large matrix multiply, where
the cubic computation disguises this square cost. For small problems, though,
the O(N**2) costs are actually dominant, and this type of malloc behavior
effectively doubles them (at least). You should be able to change
Linux's malloc behavior by setting these environment variables:
Once this is done, malloc should be cheaper, but ATLAS was tuned with
the expensive malloc. Therefore, you may be able to get better small-case
performance by rerunning the crossover search with these environment variables
set (don't do this unless you are going to keep these settings whenever you
use this library). You can rerun the search from the
ATLAS/tune/blas/gemm/ARCH directory by issuing:
By default, ATLAS will use a very low-res CPU timer. If you are installing
on a fairly unloaded machine, you can improve your install enormously by using
ATLAS's cycle-accurate walltimer. If you are using gcc as your C interface
compiler, enable the cycle-accurate walltimer by passing these flags to
configure:
If your C interface compiler is MSVC++ or Intel's C compiler, you can
still improve the accuracy of timings on an unloaded machine by passing
the flags:
For the Intel compilers, the C compiler (icl) seemed to work
out-of-the-box once I had done the MSVC++ environment variables setup
outlined above. To get the Fortran compiler (ifort) rolling,
I had to additinally add:
These same changes can be made to /usr/X11R6/bin/starxwin.bat
if you use that batchfile to start xterms instead of cygwin windows.
If you need to set up the environment after bash has already started running,
things are a little more difficult. Take
this file, modify
the paths to match your system, and then source it from the bash prompt.
In order to use ifort for the F77 interfaces, add the flag
In order to compile the C interface routines with CL (the command
line interface to MSVC++), add the flag:
It may be important to have a library that you can link without linking to
libcygwin.a, which is a GPL (not LGPL) library. To say that
you want to avoid this library, pass the following flag to configure:
Back to Window compiler overview
Some of the time, depending on how your programming environment is setup,
you also don't need to change anything to use ifort as your
linker. Occasionally,
in order to link using the using ifort, with the rest of the library
(including the interface routines) compiled with gcc, you need to
explicitly link
to both gcc's libgcc.a (which uses the LGPL, not GPL, license),
and MSVC's libcmt.lib. Unfortunately, the spaces in the typical
windows paths pose a challenge when called from the commandline, and so you
typically need to copy the file over to the cygwin directory (logical links
don't work). Therefore, before modifying or Make.inc, issue a
command like (adjust the paths to your system):
Wait until the ATLAS build is complete before modifying the Make.inc to point
at these libraries (but you must make these changes before doing
make check or make time). First, try to do a
make check before trying these mods, as you may not need to
make them! If this fails,
set your LIBS flag in Make.inc
something like (modifying the paths as necessary):
Back to Window compiler overview
The reason ATLAS presently requires gcc to get you threaded support is that
the supported Windows compilers don't supply the required include file
pthread.h. My MSVC++ is version 6, and I suspect newer versions
actually do provide pthreads. If they do, you can try to install, and
it may work. If it does, however, you will still have a small problem.
I am aware of no wall-timer under Windows, and you need a wall-timer to
get meaningful timings of threaded code. ATLAS defaults to calling a
cpu timer, which will make the install work, but will not give you
any idea how fast your lib is. Therefore, you will need to
turn on ATLAS's assembler timer.
To do this, before you do 'make install', Change line 110 of
ATLAS/makes/Make.sysinfo from:
Now, before install, but after config, edit the Make.ARCH created
by config, and add the definition -DPentiumCPS=[MyMhz] to the
CDEFS macro. For instance, on my 2.5Ghz P4, I would change the line:
Real errors have residuals very large residuals (eg., 10e15). However, even
these kinds of errors may be a result of the LAPACK tester being completely
tuned to the LAPACK BLAS implementation. For instance, the DGER operation
is supposed to do A += alpha * x * y. The reference BLAS perform
A += x*(alpha*y). ATLAS does this or A += (alpha*x)*y,
whichever is cheaper. This causes is the LIN testers to fail quite
a few residual checks with size 10e15, even though both are legal. I have
modified ATLAS to avoid these spurious GER problems, but not all of the
LAPACK testers' reliance on fixed orderings can be fixed.
Basic compiler information
ATLAS has support for one or more compilers for every platform. In general,
we provide gcc/gfortran for most supported architectures, since these compilers
are freely available. My system doesn't have the -f option to cp
If you take the following line, and put it in a file cp you make
executable, and then put it in your path before your system cp, it should
get rid of the -f option:
/bin/cp `echo $* | sed -e 's/-f / /'`
Building a complete LAPACK library
ATLAS does not provide a full LAPACK library. However, there is a simple way
to get ATLAS to provide its faster LAPACK routines to a full LAPACK library
provided by netlib. First, install lapack from netlib.
First, download and install the standard LAPACK library from the
LAPACK homepage.
Then when doing
the configure step to ATLAS pass the following flag:
--with-netlib-lapack=[path/to/lapack/lib]
mkdir tmp
cd tmp
ar x ../liblapack.a
cp <your LAPACK path & lib> ../liblapack.a
ar r ../liblapack.a *.o
cd ..
rm -rf tmp
How do I restart an install from scratch?
Simply do a rm -rf * in your BLDdir directory,
and then reconfigure and build.
How do I restart an interrupted install?
If your ATLAS install was interrupted, and you have fixed the problem,
you can usually safely (there are always exceptions; if the install died
in the middle of an ar command, for instance, many systems cannot recover)
restart the install by:
How do I do I get rid of all the .o's?
Once you have done make install (and/or manually copied the
libraries and include files you want), you can simply delete the
entire OBJdir directory.
Do NOT use the -fno-f2c flag with g77
Haven't tracked this down in a while, but it appears to break quite
a few things in fairly non-obvious ways for mixed g77/gcc libs.
What happens if I install with no Fortran compiler?
You enable this by passing --nof77 to configure. In this case,
ATLAS will still install correctly, but it will obviously not create the
Fortran77 interface libraries. You will not be able to run the
testers under the BLDdir/interfaces/ directory, since these testers
are written in Fortran. Further, ATLAS expects that you will be comparing
against a Fortran77 interface BLAS, and this will obviously not be the
case, and so you will need to make the following changes if you want to
run any of the ATLAS tester/timers, even the ones written in C:
#define USE_F77_BLAS
to:
#define USE_L1_REFERENCE
#define USE_F77_BLAS
to:
#define USE_L2_REFERENCE
#define USE_F77_BLAS
to:
#define USE_L3_REFERENCE
#define TRUST_SMALL
My performance drops off for very large problem (N > 1500)
Note this is for the serial interface. If it is only your parallel
performance that drops off for large problem sizes, you are probably
experiencing this problem.
Post install tuning.
Here are some tips to improving ATLAS performance after an install:
Tuning CacheEdge.
CacheEdge is an Level 2 Cache blocking parameter; because it's effects are
fairly subtle on most machines, it often goes wrong on machines experiencing
any kind of load, causing performance to be be suboptimal. CacheEdge can
improve performance by as much as 15%, and it can reduce ATLAS's
memory usage as well.
./xdfindCE -m [N] -n [N] -k [N]
where [N} is replaced by a very large number that is a multiple
of your blocking factor. You want to make this number as large as you
can stand to wait on, and this varies a great deal from machine to machine.
A good guestimate for most machines might be around 2000.
TA TB M N K alpha beta CacheEdge TIME MFLOPS
== == ====== ====== ====== ====== ====== ========= ========= ========
T N 1000 1000 1000 1.00 1.00 0 5.470 365.63
T N 1000 1000 1000 1.00 1.00 16 5.470 365.63
T N 1000 1000 1000 1.00 1.00 32 5.460 366.30
T N 1000 1000 1000 1.00 1.00 64 5.470 365.63
T N 1000 1000 1000 1.00 1.00 128 5.260 380.23
T N 1000 1000 1000 1.00 1.00 256 5.240 381.68
Initial CE=256KB, mflop=381.68
Best CE=256KB, mflop=381.68
So we want to set CacheEdge to 1024*256 = 262144. atlas_cacheedge
will look something like:
#ifndef ATLAS_CACHEEDGE_H
#define ATLAS_CACHEEDGE_H
#define CacheEdge 196608
#endif
If your initial install did not use CacheEdge, line 3 will be missing
completely. If you don't have this line, you would simply add it, using
the new value of 262144. In the above example, we would simply
replace 196608 with 262144.Special hints for setting CacheEdge for multiprocessor machines
CacheEdge turns out to be very important to threaded performance.
Unfortunately most of the default CacheEdge settings were obtained on
single processor machines. So, you may well be able to see a substantial
speedup by changing CacheEdge for your multiprocessor system.Changing ATLAS's maximum buffer space.
Another way to tune ATLAS to your system is to vary the amount of
buffer space ATLAS is allowed to allocate. In general, you want to
set this as high as you can without causing swapping. If you have
a machine with low memory, and you see dramatic slowdowns as the
problem size goes up, you should definitely choose a smaller max size.
ATLAS defaults to 4MB (except on the x86-64, where we default to 8MB). To
vary this value, edit the file:
ATLAS/include/atlas_lvl3.h
And set ATL_MaxMalloc to the maximal number of bytes that you want
ATLAS to allocate internally. After editing, rebuild the libs by issuing:
make xdl3blastst xcl3blastst xzl3blastst xsl3blastst
in your ATLAS/bin/ARCH directory.
Improving ATLAS small case performance by changing
malloc behavior
ATLAS allocates a buffer space for most GEMM calls. When I wrote it,
my assumption was that only first call requires a switch to kernel space
to do the allocation, and incurs the unneeded overhead of zeroing out the
memory. However, by default Linux (as well as some other OSes, such as
OS X) allocates non-trivial sized allocations using mmap, which
means that when free is called, the memory is immediately returned
to the system. Thus all malloc calls have extremely high overheads.
setenv MALLOC_TRIM_THRESHOLD_ -1
setenv MALLOC_MMAP_MAX_ 0
make sRun_tfc pre=s
make dRun_tfc pre=d
make cRun_tfc pre=c
make zRun_tfc pre=z
This search takes a *loooong* time, then to build the changes into the
libraries, go to ATLAS/bin/ARCH, and issue:
make xsl3blastst
make xdl3blastst
make xcl3blastst
make xzl3blastst
When linking ATLAS's testers, I'm getting a bunch of
undefined BLAS symbols (eg. dgemm_, dgemv_, etc).
The ATLAS BLAS testers (x[s,d,c,z]l[1,2,3]blastst) expect to compare
against a F77 interface BLAS library for performance and testing purposes.
You get these missing symbols when your Make.ARCH's BLASlib
is left blank, or does not point at a complete BLAS library. If you have
a non-ATLAS BLAS built somewhere, point the BLASlib macro at it. If you
don't, probably the easiest fix is probably to grab the
Fortran77 reference BLAS
tarfile, and build it into the required lib. If you don't want to
do this, or don't have access to Fortran77, then you can have ATLAS
test against its own C reference as discussed
here.I'm linking with C, and getting missing symbols
(such as w_wsfe, do_fio, w_esfe or
s_stop).
These kinds of symbols are Fortran library calls. The problem is that the
C linker does not automatically find the Fortran libraries. The most
common fix is to either link using your fortran linker, or to rewrite your
code so that Fortran routines are not called. If you know where they are,
you can also choose to link in the Fortran libraries explicitly.
Installing gcc under unix without being root
You do not need to be root to install a gcc that will deliver decent
performance for ATLAS. I include below the exact steps I use to install
the C compiler only in my own home area. Changing my home area path (given
in the --prefix command to configure) to yours should allow you to do the same.
These directions are for x86 users, where ATLAS needs gcc 4.2.x for decent
performance. They work pretty much the same for other platforms/gcc versions.
Note that these directions will install gfortran as well. The fortran
compiler is not needed for ATLAS performance, so if you want to use a
different fortran compiler than this version of gfortran,
simply omit fortran from the --enable-languages step.
NOTE: if your old gcc version is a lot older than the new one, there
will often by library incompatabilities, which can cause linking,
particularly of FORTRAN codes, to fail. If this occurs, you can
either use the old compiler to do linking, or set the environment
variable LD_LIBRARY_PATH so that your new libraries are seen
before the system libraries. For instance, on the above install where
the system libraries were for gcc 4.1.0, I had to add the following
line to my .cshrc:
setenv LD_LIBRARY_PATH /home/whaley/local/gcc-4.2.1/lib:/home/whaley/local/gcc-4.2.1/lib64
Help for building ATLAS under windows
You must install ATLAS in an directory without spaces in the path. Eg.,
/home/Administrator/ATLAS is fine, but /cygwin/c/My Home Area
is not.
-D c -DPentiumCPS=[your Mhz]
Obviously, [Your Mhz] should be replaced by the speed in MegaHertz
of your machine (eg. I passed -D c -DPentiumCPS=2200 to configure
on my 2.2Ghz Athlon-64).
-D c -DWALL
Building ATLAS with a non-cygwin compiler
If you want to build ATLAS with a non-cygwin compiler (i.e., a native windows
compiler such as Intel's icl or ifort), you will need
to perform the following steps (you can skip this if using the gnu compilers
is OK):
You will also need to install as root, as ATLAS builds unix-workalike compiler
wrappers and puts them in /usr/local/bin.
Setting your Windows compiler environment correctly
The easiest way to get your environment variables setup is to modify
your /cygdrive/c/cygwin/cygwin.bat file. In order to be able
to call MSVC++ (cl) from cygwin, I added the following lines
to my cygwin.bat:
chdir c:\Program Files\Microsoft Visual Studio 8\VC
call vcvarsall
Obviously, adapt the path as necessary to point to your MSVC++ install.
You can use the 'find files' function to search for the vcvarsall
file.
chdir c:\Program Files\Intel\Compiler\Fortran\10.0.025\IA32\Bin
call ifortvars
to my cygwin.bat file. In all, here is the
/cygdrive/c/cygwin/cygwin.bat that allows me to use MSVC++, icl,
and ifort:
@echo off
chdir c:\Program Files\Microsoft Visual Studio 8\VC
call vcvarsall
chdir c:\Program Files\Intel\Compiler\Fortran\10.0.025\IA32\Bin
call ifortvars
C:
chdir C:\cygwin\bin
bash --login -i
Back to Window compiler overview
Telling config about your windows compilers
Presently, ATLAS allows you to change only the interfaces compilers with
the native Windows compilers (the bulk of the library must be compiled with
gcc).
-C if ifort
to your configure command.
-C ic cl
-Si nocygwin 1
Note that without the cygwin library, you cannot build the threaded BLAS,
since Windows does not provide POSIX threads compliance, so only throw this
flag if you need it.
Post-config Make.inc fiddling for windows compilers
If you are compiling with all gnu compilers, no changes need to be made.
cp /cygdrive/c/Program\ Files/Microsoft\ Visual\ Studio\ 8/VC/lib/libcmt.lib /usr/local/lib/.
LIBS = /cygdrive/c/cygwin/usr/local/lib/libcmt.lib \
/cygdrive/c/cygwin/lib/libcygwin.a
If you built without cygwin, then the line you want is:
LIBS = /cygdrive/c/cygwin/usr/local/lib/libcmt.lib \
/cygdrive/c/cygwin/lib/gcc/i686-pc-mingw32/3.4.4/libgcc.a
and then the ATLAS timers and testers should work. You will probably
need to link
to these libraries yourself when linking to ATLAS externally as well.
Using pthreads under Windows
Right now, the only way to use pthreads under Windows is if you compile
with gcc. Due to the aforementioned gcc bug, this currently means that your
Fortran compiler must be g77. If this is acceptable, no extra steps are
needed: just run config as normal, and say you want to use threads.
$(CC) -c $(CCFLAGS) ATL_walltime.c
to:
$(BC) -c $(BCFLAGS) ATL_walltime.c
CDEFS = $(L2SIZE) $(INCLUDES) $(F2CDEFS) $(ARCHDEFS)
to:
CDEFS = $(L2SIZE) $(INCLUDES) $(F2CDEFS) $(ARCHDEFS) -DPentiumCPS=2500
How about C++ header files for the C interfaces?
Since ATLAS does not provide full OO C++ interfaces, I am reluctant to raise
the expectation that it does by providing C++ specific header files. What
I have always envisioned is the C++ programmer creating his own include
files, such as:
>cat cppblas.h
extern "C" {
#include cblas.h
}
>cat cpplapack.h
extern "C" {
#include clapack.h
}
If you are a C++ programmer using ATLAS, and think differently, let me know.
Problems with linking/missing
LAPACK routines on OS X
OS X has a built-in version of ATLAS, and uses the standard names for them.
They may be less up-to-date and/or have less libs than something you install
yourself; in particular, if you have a Fortran compiler, you can build
a full lapack library, which Apple does not currently provide, and so
many users want to install the standard ATLAS.
Unfortunately, when searching for libs the compiler looks in the
system areas where apple keeps its ATLAS libs before looking in
directories supplied by -L. This means that if you use
-L and -l for your linking, you always get Apple's modified
ATLAS, rather than the one you installed. There are two fixes for this
problem that I know of. First, you
can just link to the full name and path, rather than using -L.
For instance, change something like:
gcc -o xtst test.c -L /home/whaley/TEST/ATLAS/build64/lib -lcblas -latlas
to:
gcc -o xtst test.c /home/whaley/TEST/ATLAS/build64/lib/libcblas.a \
/home/whaley/TEST/ATLAS/build64/lib/libatlas.a
The only other trick I'm aware of is to rename your ATLAS libraries so that
the Apple versions will not override them.
Why am I failing more LAPACK tester cases?
The LAPACK testers have been hand-tuned to work with the reference
BLAS, and thus these BLAS almost always produce the least amount of failures
(though you typically get some failures even with LAPACK's BLAS). If you
scope the output files, you can quickly get an idea of which failures are
serious by looking at the residuals. Residuals that are of size O(100) are
typically not real failures, but merely a result of differing order of flops
from what the ref blas do (which is legal and expected). In fact, on many
platforms ATLAS achieves noticably less error than the reference BLAS,
but since the tester has been so heavily tuned to the reference BLAS, these
more accurate results cause more failures. You can see a simple example of
this on most x86 platforms by compiling the reference BLAS with SSE only,
which results in 64/32-bit precision. Then compile the same BLAS to use the
x87 unit (which has 80-bit precision, though the gcc or the code can
sometimes drop back to 64/32-bit precision briefly), and you will find you
have more errors, despite having at least the same precision in all cases,
and usually much more precision.
Installation help for AIX
Under AIX, you need to set the environment variable OBJECT_MODE
appropriately. So, if you want a 64-bit library and you use the bash shell,
you would do export OBJECT_MODE=64, while
setenv OBJECT_MODE 32 would be for 32-bit libraries under
csh.