institutional_challenges.tex

\label{sec:institutional-challenges}
Many adaptations in software engineering for scientific applications
described in the previous section pertain to software design and
testing. A number of challenges also arise because of the kind of
organizations and the research communities where these codes are
developed. The most crippling and pervasive 
challenge faced by scientific codes in general, and multi-physics codes in
particular, is that funding for software development and maintenance is difficult to attain. 
There is evidence that when software is designed well
it pays huge dividends in scientific productivity from the
small number of projects that secured such funding for software
infrastructure design. Examples include community codes such as NAMD
\cite{phillips2005scalable}, Amber \cite{case2014amber} and Enzo \cite{Enzo2013} which are used by
significant number of users in their respective communities. More
persuasive case can be made by a handful of codes such as FLASH \cite{Dubey2009, Dubey2008},
Cactus \cite{blazewiczphysics} and Uintah \cite{TGRID10,uintah2} that were built for one community,
but have since expanded their capabilities to serve several other
communities using a common infrastructure.

Even with this evidence it remains difficult to obtain
funding for investment in software engineering best practices.
Available funding is most often carved out of scientific goal 
oriented projects that have their own priorities and time-line. This
model often ends up short-changing the software engineering.
The scientific output of applications is measured in terms of publications, which in
turn depend upon data produced by the simulations. Therefore, in a
project driven purely by scientific objectives, the short-term science
goals can lead to situations where quick-and-dirty triumphs over
long term planning and design. The cost of future lost productivity
may not be appreciated until much later when code base has  grown too
large to remove its deficiencies in any easy way.  Software
engineering is forcibly imposed on the code, which is at best a
band-aid solution. 

Another institutional challenge in developing good software
engineering practices for scientific codes is training students and staff to
use the application properly. Multi-physics codes require a broad range
of expertise in domain science from their developers, and software
engineering skills is an added requirement.  Often experts in a domain
science who develop scientific codes are not trained in software engineering
and many learn skills on the job through reading, or talking to
colleagues \cite{hannay2009,Nguyen-Hoan}. Practices are applied as they understand them, usually
picking only what is of most importance for their own development.
This can be both good and bad. Good because it sifts out the unnecessary aspects of SE
practice, and bad because it is not always true that the sifted out
aspects were really not necessary. It might just be that the person
adopting the practice did not understand the usefulness and impact of
those aspects.

Institutional challenges also arise from scarcity and stability of
resources apart from funding. The domain and numerical algorithmic
expertise is rarely replicated in a team developing the multi-physics
scientific application.  Even otherwise, deep expertise in the domain may be
needed to model the phenomenon right, and that kind of expertise is
relatively rare. Then there is the challenge of communicating the
model to the software engineer, if there is one on the team, or to
team members with some other domain expertise. It requires at least a
few developers in the team who can act as interpreters for various
domain expertise and are able to integrate them. Such abilities take a
lot of time and effort to develop, neither of which are
easy in academic institutions where these codes are typically
organically grown. The available human resources in these institutions
are post-docs and students who move on, so there is little retention of
institutional knowledge about the code.  A few projects that do see
the need for software professionals struggle to find ways of funding
them or providing a path for their professional growth. 

The above institutional challenges are among the reasons why it is
hard and often even undesirable to adopt any set software development
methodology in scientific application projects. For example, the principles
behind the agile manifesto apply, but not all the formalized processes
do. Agile software methods \cite{agile} are lightweight evolutionary development
methods with focus on adaptability and flexibility, as opposed to
waterfall methods which are sequential development processes where
progress is perceived as a downward flow \cite{waterfall}. 
Agile methods aim to deliver working software as early as possible
within the lifecycle and improve it based upon user feedback and
changing needs. These aims fit well with the objectives of scientific
software development as well. These codes are developed by
interdisciplinary teams where interactions and collaborations are
preferred over regimented process. The code is simultaneously
developed and used for science, so that when requirements change there
is quick feedback.  For the  same reason, the code needs to be in
working condition almost all the time. However, scarcity of resources
does not allow the professional roles in the agile process to be
played out efficiently.  There is no clear separation between the
developer and the client, many developers of the code are also
scientists who use it for their research.  Because software
development goes hand-in-hand with research and exploration of
algorithms, it is impossible to do either within fixed
time-frames. This constraint effectively eliminates using agile
methods such as {\em sprints} or {\em extreme programming}
\cite{carver2007software}. The waterfall model is even less useful
because it is not cost-effective or even possible to have a full
specification ahead of time. The code has to grow and alter
organically as the scientific understanding grows, the effect of using
technologies are digested and requirements change. A reasonable
solution is to adopt those elements of the methodologies that match
the needs and objectives of the team, adjust them where needed, and
develop their own processes and methodologies where none of the
available options apply.   

Due to the need for deep expertise, and the fact that the developer of a
complex physics module is almost definitely going to leave with
possibly no replacement, documentation of various kind takes on a
crucial role. It becomes necessary to document the algorithm, the
implementation choices, and the range of operation. The generally
preferred practice of writing self explanatory code helps, but does
not suffice. To an expert in the field, who has comprehensive
understanding of the underlying math, such a code might be accessible
without inline documentation. But not to non-experts (i.e. from
another field or a software engineer in the team if there is one) who
may have reasons to look at the code. For longevity and
extensibility, a scientific code must have inline documentation
explaining the implementation logic, and reasons behind the
choices made.   

\subsection*{Key Insights}
\label{institutional-insights}
\begin{itemize}
\item The benefits of investment in software design or process are not
appreciated, and the funding model is not helpful in promoting them either. 
\item Development requires interdisciplinary teams with good
communication, which is difficult in academic institutions. 
\item Methodologies get better foothold if they are flexible and adapt
to the needs of the development team
\item Developer population is transient, detailed inlined documentation is
necessary for maintenance.
\end{itemize}