1.1 --- a/0__Papers/Holistic_Model/Perf_Tune__long_version_for_TACO/latex/Holistic_Perf_Tuning.tex	Sat Aug 03 19:24:22 2013 -0700
     1.2 +++ b/0__Papers/Holistic_Model/Perf_Tune__long_version_for_TACO/latex/Holistic_Perf_Tuning.tex	Fri Sep 13 11:02:18 2013 -0700
     1.3 @@ -57,7 +57,7 @@
     1.4  \begin{document}
     1.5  
     1.6  %MOIRAI: MOdel for Integrated Runtime Analysis through Instrumentation
     1.7 -\title{An Approach to Augment Performance Tuning\\ via Visualization of Causality Chains}
     1.8 +\title{Augmenting Performance Tuning\\ via Visualization of Cause-and-Effect Chains}
     1.9  
    1.10  %\authorinfo{Nina Engelhardt}
    1.11  %           {TU Berlin}
    1.12 @@ -79,100 +79,168 @@
    1.13  
    1.14  
    1.15  \begin{abstract}
    1.16 -Performance tuning is an important aspect of parallel programming that involves understanding both sequential behavior inside work-units and scheduling related behavior arising from how those units are assigned to hardware resources. Many good tools exist for identifying hot-spots in code, and tuning the sequential aspects there, as well as the data layout. They also identify constructs that spend a  long time blocked. However, they provide less help in understanding  why  constructs block for so long, and why no other work fills in.   The answer      often arises from chains of  causal interactions that involve the scheduling choices made in combination with runtime implementation details and  hardware behaviors. Identifying such a chain   requires visualizing each source of causality.  We propose supplementing existing tools with an additional tool to identify these chains and tie the behaviors back to specific spots in the code. To make such a tool, we apply a newly discovered model of the structure of parallel computation, as a guide to gathering each step in the scheduling and execution process. The visualizations of these are superficially similar to existing tools, but include additional  features  to identify unexpected behaviors within the chain and tie them to  related parallelism constructs within the code.
    1.17 -We show how to instrument a parallel language or programming model,   with no need to modify the application.  To simplify illustration, we instrumented the runtime of our own  pi-calculus inspired programming model, called SSR, and we walk through a tuning session on a large multi-core machine, which demonstrates the improvements in  generating hypotheses for both the causes of idle cores, and how to reduce the idleness. 
    1.18 +Performance tuning is an important aspect of parallel programming. It involves  tuning both sequential behavior, inside units of work, and also tuning the scheduling of those units onto hardware resources. Many good tools exist for the sequential aspects such as finding hot spots and adjusting the data layout. Many also show statistics related to scheduling, such as periods of idleness on each core and time spent trying to acquire locks. Some exceptional tools even shift blame onto specific parallelism construct instances in the code. However, more help is often desired, to understand the cause-and-effect among the events leading to the pattern of idleness.  Such cause-and-effect tells the coder why the blamed constructs consume so much time, and so enables forming a hypothesis for how to improve the code.  We propose an additional tool that visualizes the cause-and-effect cascades. It relies upon a novel theory of the structure of parallel computation, which we touch upon. The resulting visualizations appear superficially similar to existing tools, but visual details convey extra information, to reveal the cascades and  relate  them to the parallelism constructs within the code. The application is not modified, only the runtime system of the parallel language  or programming model.  We present a tutorial style introduction to the theory and the tool.  We instrumented the runtime of our own  pi-calculus inspired programming model, called SSR, and walk through using it to  read the cascade of causes of idle cores, and form hypotheses for how to reduce the idleness. 
    1.19  \end{abstract}
    1.20  
    1.21  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    1.22  \section{Introduction}
    1.23  \label{sec:intro}
    1.24  
    1.25 -Performance visualization and tuning tools for parallel programs are critical to achieving good performance.
    1.26 -Large factors of improvement can be realized by using existing tools to identify hot-spots and adjust the data layout and other internals of the most time-consuming functions.  
    1.27 -
    1.28 -
    1.29 -
    1.30 -However, once such optimization has made progress, the main area available for performance improvement undergoes a shift towards the scheduling  choices made by the runtime. Existing  tools tend to provide less help here, in understanding the complicated interactions among runtime behavior, scheduling decisions, and consequent hardware behavior. The interactions often give rise to unexpected chain reactions that leave cores idle. These involve the constraints stated by the application, the way the work is divided, and the choices made during the run for which unit of work to assign to which hardware resource at which point in time. To tune such behaviors, the chains need to be identified and understood.   
    1.31 +Performance visualization and tuning tools for parallel programs are critical to achieving good performance. Existing tools can be used to achieve large factors of improvement. The first way is through  identifying hot-spots and adjusting the data layout and other internals of the most time-consuming functions. This type of tuning considers mainly the sequential aspects of the code.  
    1.32 +
    1.33 +Once such optimization has made progress, further improvement shifts towards the scheduling aspects, which include the choices for which work is placed where.  This involves the internals of the runtime system. Improvement here relies on understanding the complicated interactions among runtime behavior, scheduling decisions, and consequent hardware behavior.  Unexpected performance losses often involve chain reactions that leave cores idle. The chains involve the constraints stated by the application, the way the work is divided, and the choices made during the run for which unit of work to assign to which hardware resource at which point in time. To tune such behaviors, the chains need to be revealed, in order to understand the causes and effects involved.   
    1.34  
    1.35  %The level of complexity is reflected by the fact that finding the optimal set of scheduling choices is known to be an  NP hard problem [].   
    1.36  
    1.37 -For example, a unit of work completes, which sends a signal to the runtime to update the state of the unit and the state of the hardware resource it occupied. This in turn causes the runtime to choose a different unit to own that hardware and sends the meta-information for that unit to the hardware. This in turn triggers communication, to move  the data consumed by the unit  to the hardware. Then the work of the new unit takes place there.
    1.38 -
    1.39 -Any one of these interactions could be individually abnormal, and an unexpected source of performance loss. For example, there may be congestion in the network that causes the data movement to be especially slow for that particular new unit. 
    1.40 -
    1.41 -Although current tools do a good job of identifying which constructs block, and which cores sit idle, they don't help much in deciphering such complex chains of causal interactions. This needs a visualization of each step in the causal chain.
    1.42 -
    1.43 -
    1.44 -
    1.45 -    
    1.46 -
    1.47 -
    1.48 -To help with this particular aspect of tuning, we propose an additional tool. It collects the causal interactions inside the runtime and hardware, and then visualizes them in a useful way. In addition, it helps to generate ideas for ways to reduce idleness by showing the scheduling-related structure of the application.
    1.49 -
    1.50 -  
    1.51 -
    1.52 -Our approach  is based on  a newly discovered model of the structure of parallel computation, which identifies each causal step. 
    1.53 -The theoretical model adds value   by indicating particular quantities to measure at specific points in the runtime system and by establishing a mental framework that simplifies hypothesis generation.
    1.54 -
    1.55 -The information used during tuning is collected by the runtime as the application executes, which makes it practical and backwards compatible. No modification of applications takes place.
    1.56 -
    1.57 -The information is used to visually link each unit to the units upon which its execution is conditional, via a chain of causal steps.   Semantics attached to the visual features  enable quickly generating the correct hypotheses for the causes of  lost computation opportunities, and quickly narrowing down what can be done, in the application code, to improve performance. In effect, the visualization serves as a map showing idle cores linked back to the sections of application code related.
    1.58 -
    1.59 -
    1.60 -To simplify seeing the value of the approach,  we walk through a session of tuning scheduling choices, in which we point out the semantics of features in the visualization, what each visual feature implies, and how these implications lead to hypotheses and fixes.   Although the views look similar to current tools, they differ in the semantics of the visual features.
    1.61 -
    1.62 -  
    1.63 -
    1.64 -We intend the  contribution of this paper to be the way the parallel model has been applied, rather than any aspects of the model itself.  The distinction is important because the space is used to convey the value gained by the  process of applying the model. Only enough of the model is stated, in \S \ref{sec:theory} and \S \ref{sec:Implementation}, to understand  where the value comes from and how to  instrument a runtime to gain it.
    1.65 -
    1.66 -This paper is organized as follows:  we introduce the semantics of features of our visualization  by walking through a tuning session in Section \ref{sec:casestudy}, and then expand on the theoretical model behind it in Section \ref{sec:theory}. Section \ref{sec:Implementation} gives  details of instrumenting a runtime to collect data according to the model. We relate the approach to other tuning tools in Section \ref{sec:related} and draw conclusions in Section \ref{sec:conclusion}.
    1.67 -
    1.68 +As an example of such a chain, a unit of work completes, which sends a signal to the runtime to update the runtime's internal state of the unit and the state of the hardware resource it occupied. This in turn causes the runtime to choose a different unit to own that hardware and sends the meta-information for that unit to the hardware. This in turn triggers communication, to move  the data consumed by the unit  to the hardware. Then the work of the new unit takes place there.
    1.69 +
    1.70 +Any one of these interactions could be individually abnormal, and an unexpected source of performance loss. In particular, there may be congestion in the network that causes the data movement to be especially slow for that new unit, or the runtime may experience contention for access to its internal state (IE, contention for an internal lock). 
    1.71 +
    1.72 +We propose augmenting current tools with a way to visualize each link in such cause-and-effect cascades.  The augmentation focuses on just this one aspect of performance tuning.  It adds the ability to see inside the runtime system's operation, so that all links in the cause and effect chains can be visualized.  It also assigns a meaning to each link, to aid in understanding the patterns.  The meaning is determined according to a theory of parallel computation. We touch upon this theory just enough to explain the meaning assigned to each link.  
    1.73 +
    1.74 +
    1.75 +%To help with this particular aspect of tuning, we propose an additional tool. It collects the causal interactions inside the runtime and hardware, and then visualizes them in a useful way. In addition, it helps to generate ideas for ways to reduce idleness by showing the scheduling-related structure of the application.
    1.76 +
    1.77 +%Our approach  is based on  a newly discovered model of the structure of parallel computation, which identifies each causal step. The theoretical model adds value   by indicating particular quantities to measure at specific points in the runtime system and by establishing a mental framework that simplifies hypothesis generation.
    1.78 +
    1.79 +No modification of the application takes place.  The new tool instruments the language implementation.  Instrumentation is inserted at key points in the runtime system of the language.  All the information used to visualize the cause and effect cascades is collected by the runtime system as the application executes. 
    1.80 +
    1.81 +The information is presented visually, as blocks of time taken by units of work that were scheduled, and as links among those blocks of time.  The time counted against a work-unit is broken down into categories such as: time taken to decide where to execute the unit; time to move the data of the unit; time inside the runtime to communicate completion of units and changes in the states of hardware resources; and time to move the meta-information of the unit.  The categories are defined by the computation theory.
    1.82 +
    1.83 +%presented as  broken down , and links link each unit to the units upon which its execution is conditional, via a chain of causal steps.   Semantics attached to the visual features  enable quickly generating the correct hypotheses for the causes of  lost computation opportunities, and quickly narrowing down what can be done, in the application code, to improve performance. In effect, the visualization serves as a map showing idle cores linked back to the sections of application code related.
    1.84 +
    1.85 +This introductory paper focuses on explaining the basics of the approach in a tutorial fashion. This involves learning the categories defined by the theory as well as the meaning of the fine details within the visualization. It also covers learning how to generate hypotheses, from the visualization, about what to change in the code, to modify the cascades of cause-and-effect.
    1.86 +
    1.87 +This paper does not cover advanced uses of the approach, as space requires choosing between providing tutorial style understanding, versus illustrating advanced features absent understanding of what's being presented.  Expert readers may find fault with our choice, desiring a more in depth comparison to the defacto tools, illustrating what this paper's tool adds beyond them.  We ask such readers to be patient, take the time to understand our approach, and allow us to present the more advanced features in a future paper.
    1.88 +
    1.89 +
    1.90 +We will present ways the proposed tool is separated from other tools, namely by adding detail to where a core's time has been spent, and by providing a wider variety of cause-and-effect links, that even cover runtime internal causations.  At the end of the section on the basic theory, \S\ref{subsec:SCG}, we revisit describing how the proposed tool goes further, to supplement existing ones.
    1.91 +
    1.92 +
    1.93 +%To simplify seeing the value of the approach,  we walk through a session of tuning scheduling choices, in which we point out the semantics of features in the visualization, what each visual feature implies, and how these implications lead to hypotheses and fixes.   Although the views look similar to current tools, they differ in the semantics of the visual features.
    1.94 +
    1.95 +%The contribution of this paper to be the way the parallel model has been applied, rather than any aspects of the model itself.  The distinction is important because the space is used to convey the value gained by the  process of applying the model. Only enough of the model is stated, in \S \ref{sec:theory} and \S \ref{sec:Implementation}, to understand  where the value comes from and how to  instrument a runtime to gain it.
    1.96 +
    1.97 +This paper is organized as follows:  we introduce the basics of the theory behind the tool in \S\ref{sec:basics}, then give the features of our visualization and their meaning in \S\ref{sec:visualization}. We illustrate their use by walking through a tuning session in \S\ref{sec:casestudy}. To prepare for explaining how to implement the approach, we expand on the theory behind it in \S\ref{sec:theory}. \S\ref{sec:Implementation} then gives the details of how to instrument a runtime to collect the data dictated by the theory. We briefly relate the approach to other tuning tools in \S\ref{sec:related} and draw conclusions in \S\ref{sec:conclusion}.
    1.98  
    1.99  
   1.100  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   1.101 -\section{Illustration of Tuning Scheduling Decisions}
   1.102 -\label{sec:casestudy}
   1.103 -
   1.104 -To demonstrate the value of the approach, we create a slightly contrived example  that brings out particular features of the tool. but uses a real application running on real hardware. 
   1.105 -
   1.106 -We start  by describing the hardware being run on and structure of the program being tuned, and follow by describing the parallel language used and instrumented. We then  advance to the features of the visualization.
   1.107 -
   1.108 - A sequence of visualizations follow. In each,  we point out how the  performance loss is identified, and which visual features suggest the hypothesis of the cause of the loss.  We show how the visual features direct the user to the specific sections of code that need to be changed, and how the model suggests what changes to try. 
   1.109 -
   1.110 -\subsection{Setup: Hardware and Application}
   1.111 -
   1.112 -
   1.113 -
   1.114 -We run our example on a machine with 4 sockets by 10 cores each, for a total of 40 physical cores. They are Intel WestmereEx cores running at 3.0GHz, with TurboBoost turned off for reproducability. 
   1.115 -
   1.116 -We wish to tune scheduling on a standard program that the reader  knows well, so we chose matrix multiply, with which the reader should be familiar.  This allows concentration on the tool without distraction about the application. 
   1.117 -
   1.118 -The application is structured as follows: it  creates one entity to divide the work into a number of pieces and  creates another entity for each piece of work. How many pieces is determined by the combination of a tuning parameter in the code with the number of cores. The work is distributed across the cores  in a round-robin fashion, unless otherwise specified.
   1.119 -
   1.120 -The application also creates an entity that manages the partial-results. Each piece of work sends its contribution, which is accumulated into the overall  result. The entity that divides waits for the entity that accumulates to signal completion then the language runtime shuts down.
   1.121 -
   1.122 -\subsection{Programming Model}
   1.123 -We chose a simple language that was convenient to instrument. It is inspired by pi-calculus, and called Synchronous Send-Receive (SSR).  It implements  rendez-vous style send and receive operations made between virtual processors (VPs), where a VP is similar to a software thread. The example application uses the commands for creating and destroying VPs, two kinds of send-receive paired operations, a parallel singleton, and scheduling control constructs. 
   1.124 -
   1.125 -The first kind of send-receive pair used, and instrumented, is precise about sender and receiver. Called \emph{send\_from\_to}, it specifies both sender and receiver VPs, and is used by the results VP  to tell the divider VP that the work is complete. The second pair, \emph{send\_of\_type\_to}, specifies only a specific receiver, leaving the sender anonymous. It has increased flexibility while maintaining some control over scope. The worker VPs use this to send their partial result to the results VP. 
   1.126 -
   1.127 -The application uses the \emph{singleton} construct to reduce the amount of work done by the (sequential) divider VP. The construct designates a piece of code as to be executed only once, even though it is invoked by multiple VPs. It is employed to shift the work of copying matrix fragments out of the divider and over to the worker-pieces. The first worker-piece to use a given input-matrix fragment performs the copy, which spreads the copying across the cores.
   1.128 -
   1.129 -We control the scheduling behaviors, in order to concisely illustrate the use and features of the tool. This is done with language constructs that force which core a virtual processor is assigned to. 
   1.130 -
   1.131 -A note on terminology: We often use the term  ``work-unit'',  which we define precisely, instead of ``task'', which has acquired multiple  meanings in the literature. Work-unit  is defined as the trace-segment performed on a core, between two successive scheduling events, plus the set of datums consumed by that trace segment. The word task often maps well onto this  definition, and we use both words, but mean the precise work-unit definition when we say task.
   1.132 -
   1.133 -
   1.134 +\section{Basics of the Approach}
   1.135 +\label{sec:basics}
   1.136 +
   1.137 +learning the categories defined by the theory
   1.138 +
   1.139 +as well as the meaning of the fine details within the visualization.
   1.140 +we introduce the features of our visualization and their meaning
   1.141 +
   1.142 +\subsection{Elements of the Theory}
   1.143 +The core purpose of the theory of parallel computation that we base the tool on is to connect the contributions of application code, language implementation, and hardware.  It is meant to first establish a set of primitive concepts that form the basis of parallel computation, and then establish relationships among those primitives.  The user of the theory can then look at application code, the language implementation and the hardware, and understand how the three interact. The interaction says how observed performance came about, or can predict performance while consuming minimal computation. Such understanding of the interaction can be useful in many ways.  The tool described in this paper is just one useful way to apply the theory. 
   1.144 +
   1.145 +\subsubsection{Primitive Elements of the Theory}
   1.146 +The theory chooses different primitives than are used in the broad literature and common practice.  It doesn't consider the concept of compiler to be well defined, nor the concept of runtime system, nor even the concept of programming language.  Yet, these things all, clearly are concepts in common usage and each plays a role in the performance of parallel code.  So, the theory attempts to identify patterns within those that are, indeed, well defined and also remain invariant across applications, languages, and tool chains.  Such a rearrangement of basic terms comes at the price of a steep learning curve, making the theory challenging to accept.
   1.147 +
   1.148 +The primitives chosen are: unit-of-work, constraints on scheduling a unit of work, managing the constraints, and mapping free units of work onto animators that perform the work.  These primitives can then be related to compilers, programming languages, runtime systems, and so forth.  Each of those things can be defined by the effect they have on the primitives of the theory.  We leave such definitions to other papers, while here simply asserting that this set of primitives appears to be universal, invariant, and capable of being a basis set upon which essentially all aspects of parallel computation can be understood.  We ask the patience of the reader to accept this rather bold claim for the moment, and examine it more closely in the context of future papers focused on such claims.
   1.149 +
   1.150 +These primitives are defined in terms of each other, so their defintions have a circular quality, but as a whole they form a consistent interlocking set.  Formal definitions do exist, but are outside the scope of this paper.
   1.151 +
   1.152 +\begin{description}
   1.153 +\item[A unit of work:] (AKA work unit or just unit), this is  the thing that is scheduled.  It is defined as the thing that constraints are stated about, which implies that a scheduling decision must be made for each unit of work, and that a single unit of work is indivisible from the point of view of scheduling.  Examples of work units are a single firing of a dataflow node, the trace of instructions between consecutive calls to pthread constructs, and a single iteration of a parallel-for loop. They are not always defined statically. It may only be during the run that the contents of a unit become determined.
   1.154 +
   1.155 +\item[A constraint on scheduling a unit of work:]Loosely, this is a  condition that must be satisfied before the unit of work can be assigned to an animator. Examples include "must acquire the lock variable", "must wait for a paired send from a different animator", and "comes after completion of the preceding work-unit".
   1.156 +
   1.157 +\item[Managing constraints:] This, in short is part of scheduling. It is the process of communicating among the animators the changes, in states of units and in internal constraint state, plus computing on those such that all constraints on scheduling work are upheld during the evolution of a computation.  Some languages place the bulk of this management inside the compiler, others place the bulk inside a runtime system.
   1.158 +
   1.159 +\item[Mapping free units of work onto animators:] This, in short is assignment of work onto cores.  This takes place after constraint management has communicated that particular units are free from constraints.  Sometimes this mapping is performed statically inside the compiler, sometimes it is performed as part of application code, and sometimes it is performed inside a language's or execution model's runtime system. This is the other significant part of the scheduling process. 
   1.160 +
   1.161 +\end{description}
   1.162 +
   1.163 +\subsection{Organizing information related to the primitives}
   1.164 +To organize information related to those primitive elements, the theory defines two kinds of graph.  The graphs encode information for a given application, written in terms of a given language, and executed on given hardware.
   1.165 +
   1.166 +The first kind of graph, called a Unit and Constraint Collection (UCC), encodes the units of work and the constraints on scheduling them.  It essentially characterizes the parallelism in the application, although \S\ref{sec:UCC} shows that, in the general case, the units and constraints are determined over the course of a run, by more than just the application code. A concrete UCC captures all of the units and constraints for a given run.
   1.167 +
   1.168 +The second kind of graph, called a Scheduling Consequence Graph (SCG), is where the cause-and-effect cascades are represented. An SCG also depicts the work units, but instead of constraints, it encodes the particular scheduling choices, made during a given run on given hardware.  It accounts for all time taken by each processing core, dividing it among categories and charging it against particular work units. It then connects the individual bits of time to any other bits that have a cause-and-effect relationship. 
   1.169 +
   1.170 +The extended capabilities of the tool are due to the kinds of categories it can account for,  and the ability to connect these bits together in  cause-and-effect relationships.
   1.171 + 
   1.172 +\subsection{UCC}
   1.173 +
   1.174 +For the Unit \& Constraint Collection, or UCC, the elements are the application's units of work  and the constraints on scheduling them. Constraints can be explicitly stated in the code, such as ``acquire lock", or  implied by language constructs, such as ``parallel-for''. They limit the choices available to the runtime. We use the  general term ``constraint'' instead of the specific term ``dependency'' because dependency only covers  one pattern: this unit before that one. Meanwhile  constraints are general,  such as e.g. a mutual exclusion on scheduling a group of units: any order, but only one at a time.
   1.175 +
   1.176 +
   1.177 +\subsection{The SCG, and categories of time}
   1.178 +\label{subsec:SCG}
   1.179 +The consequence graph shows the actual scheduling choices \emph{made}. The UCC defines which choices are \emph{allowed}, then from among those, a consequence graph depicts \emph{one} set of choices taken, along with the time consequences.
   1.180 +
   1.181 +Each bit of core time is accounted to a category, then added to the total for a particular unit of work.  This is depicted as boxes, one box for each unit, with a region inside the box for each category of time.  
   1.182 +
   1.183 +There is a category for each of the theory's primitive elements: creating a unit of work, adding constraints on a unit of work, managing the constraints, and mapping free units of work onto animators.  There are additional categories implied by those: performing the work, waiting idly for communication of work data,  waiting idly to receive a unit to animate, and doing internal runtime activities. 
   1.184 +
   1.185 +The internal runtime activities are always related to one of the primitive elements of the theory, but the structure of the runtime shifts which of the activities are significant enough to warrant separate accounting.
   1.186 +  Some examples include time spent communicating constraint updates from one core to another, time spent sending the meta-information about a work unit between cores, and time acquiring locks protecting internal runtime state such as shared constraint state.  On a shared memory machine, the communication times may be negligible and "rolled into" the management category, while on a distributed memory machine this communication may be a significant portion of the runtime overhead.  An SCG comes together with a definition of the internal runtime activities that it accounts to each region.
   1.187 +
   1.188 +Cause-and-effect exists between regions of different units.  These are depicted as arcs that link regions, or link entire boxes.  A region may source or sink multiple causality links. Each kind of cause and effect is represented by a corresponding kind of link.
   1.189 +
   1.190 +Groupings of kinds of causal links are defined, for convenience: constraint related cause-and-effect, runtime internal cause-and-effect, and hardware related cause-and-effect.  Constraint related links include satisfaction of a sequential dependency in the base language, and satisfaction of a parallel constraint, such as when one unit does something to satisfy a constraint on the other, causing it to be free to be scheduled. An internal runtime link example is when the runtime on one core releases an internal lock, causing the other core to acquire it. An example of a hardware causality is when one work-unit finishes on a core, it frees it, causing the runtime to calculate which work-unit to start there next.
   1.191 +
   1.192 +The proposed tool is separated from other tools by the presence of the regions, which add detail to where a core's time has been spent, and by the cause-and-effect links that cover even runtime internal causations.  Some tools, notably ones based on MPI, have the notion of unit of work, and depict time spent on each, and have some categories of cause-and-effect.  However, the proposed tool goes further by defining categories inside of a unit's time, making unit a universal concept, even for languages that don't naturally expose them, and by including additional sources of cause-and-effect, down into the runtime internals, when they are important to understand the cascades that result in idle cores.
   1.193 +
   1.194 +
   1.195 +
   1.196 +%======================
   1.197 +%%%%%%%%%%%%%%%
   1.198 +
   1.199 +%The categories are: 
   1.200 +%\begin{enumerate}
   1.201 +%\item creation of a meta-unit
   1.202 +%\item state updates that affect constraints on the unit
   1.203 +%\item the time to make the decision to animate the unit
   1.204 +%\item movement of the meta-unit plus data to physical resources that do the animation
   1.205 +%\item animation of the unit which does the work
   1.206 +%\item communication of state-updates about completion of the unit and freeing the hardware
   1.207 +%\item resulting constraint updates communicated within the runtime, possibly causing new meta-unit creations or freeing other meta-units to be chosen for animation
   1.208 +%\end{enumerate}
   1.209 +
   1.210 +%The runtime region has sub-activities, but we do not detail them here due to space. However, some will be stated in Section \ref{sec:Implementation} when we talk about instrumenting a runtime.
   1.211 +%%%%%%%%%%%%%%%
   1.212 +
   1.213 +%Arcs are gathered into groups according to the nature of the causality they represent.  The kinds of causal links are: satisfaction of a sequential dependency in the base language; satisfaction of a parallel constraint  (i.e., one unit did something to satisfy a constraint on the other, causing it to be free to be scheduled); a causal link internal to the runtime (for example, the runtime on one core releasing a shared lock, causing the other core to acquire it); and causal links in the hardware (for example, one work-unit finishes on a core, causing another work-unit to start there, modulo a choice by the runtime).
   1.214 +
   1.215 +%We will now expand on each of those kinds of causal link.
   1.216 +
   1.217 +%\paragraph{Constraint  causal link} Two entire boxes (units) are linked this way when   action by one unit contributes to satisfaction of a constraint blocking the other unit. This includes sequential dependencies from the base language (which are noted in the tool but normally not displayed).
   1.218 +
   1.219 +%Sequential dependencies may add superfluous constraints that  eliminate some otherwise allowed choices in the UCC. An example would be a sequential \texttt{for} loop that creates work-units -- no parallelism constructs cause the creations to be done in sequence, but the base C language sequentializes it nonetheless. 
   1.220 +
   1.221 +%\paragraph{Runtime internal causal link} Runtime implementation details may introduce ``extra" causalities between units. For example, the runtime we instrumented for this paper runs separately on each core and relies upon a global lock for accessing shared runtime information. This lock introduces a causal relationship  when the runtime on one core is attempting to process one unit, but must wait for the runtime on a different core to finish with its unit.
   1.222 +
   1.223 +% Normally, these are not displayed explicitly, due to clutter, but can be turned on when needed, for instance to determine the cause of a particular pattern of core usage.
   1.224 +
   1.225 +%\paragraph{Hardware causal link} The physical fact that a given resource can only be used by one work-unit at a time introduces hardware causalities. When multiple units are free to execute, but all cores are busy, then completion of a unit  on one core causes (in part) the next ready unit to run on that core. 
   1.226 +
   1.227 +%These are also not normally displayed, due to clutter, and not all hardware dependencies are directly measured. Future work will focus on using the performance counters and other instrumentation to add more information about communication paths taken as a consequence of the scheduling decisions made. It will start with the current linkage of application-code to runtime decisions, and add consequent usage of communication hardware. This gives an end-to-end linkage between runtime choices and caused behavior on the hardware. 
   1.228 +
   1.229 +%%%%%%%%%%%%%%%
   1.230 +
   1.231 +%Every unit has a meta-unit that represents it in the runtime. A  unit is defined as the work that exists after leaving the runtime, up until re-entering it. For example, the trace of instructions on a core, from the point of leaving the runtime up until the next invocation. Looking at this in more detail, every runtime has some form of internal bookkeeping state for a unit, used to track constraints on it and make decisions about when and where to execute. This exists even if that state is just a pointer to a function that sits in a queue. We call this bookkeeping state for a unit the meta-unit.
   1.232 +
   1.233 +%Each  unit also has a life-line, which progresses so:  creation of the meta-unit~\pointer~state updates that affect constraints on the unit~\pointer~the decision is made to animate the unit~ \pointer~movement of the meta-unit plus data to physical resources that do the animation~\pointer~animation of the unit, which does the work~\pointer~communication of state-update, that unit has completed, and hardware is free~\pointer~constraint updates within runtime, possibly causing new meta-unit creations or freeing other meta-units to be chosen for animation.  This repeats for each unit. Each step is part of the model.
   1.234 +
   1.235 +% Note a few implications: first, many activities internal to the runtime are part of a unit's life-line, and take place when only the meta-unit exists, before or after the work of the actual unit; second, communication that is internal to the runtime is part of the unit life-line, such as state updates; third, creation may be implied, such as in pthreads, or triggered such as in dataflow, or be by explicit command such as in StarSs. Once created, a meta-unit may languish before the unit it represents is free to be animated.
   1.236 +
   1.237 +%This explains why the visualizations remain largely the same across languages. The concepts of a meta-unit, a unit, constraints on a unit, and a unit life-line are all valid in every language.  The visualizations are based on these concepts, and so likewise largely remain the same.  In the UCC, only the constraint patterns that represent  the language's constructs change between languages. In the SCG, only which construct a line in the SCG represents changes.
   1.238 +%%%%%%%%%%%%%%
   1.239 +%==================
   1.240  
   1.241  \subsection{The Visualizations}
   1.242  \label{subsec:visualization_def}
   1.243 - The approach has two kinds of visualization, each corresponds to an aspect of the model. One focuses on just the application, conveying   its scheduling related structure. Its main value is understanding what's possible. The other focuses on how this structure interacts with the runtime and hardware. Its value is displaying the causal chains  and linking each step in a chain back to application code. 
   1.244 -
   1.245 - We refer to the first visualization  as a  Unit \& Constraint Collection, or UCC, which is explained in \S \ref{sec:UCCExpl}. Its elements are the application's units of work  and the constraints on scheduling them. Constraints can be explicitly stated in the code, or  implied by language constructs. They limit the choices available to the runtime. We use the  general term ``constraint'' instead of the specific term ``dependency'' because dependency only covers  one pattern: this unit before that one. Meanwhile  constraints are general,  such as e.g. a mutual exclusion on scheduling a group of units: any order, but only one at a time.
   1.246 -
   1.247 -We refer to the second visualization as a scheduling consequence graph (SCG), or just consequence graph, which is explained in \S \ref{sec:SCGExpl}. It depicts where the runtime assigned each of the units, how long the units executed, the sources of overhead, and the changes in constraint-state that triggered each runtime behavior. 
   1.248 -
   1.249 -In short, the UCC states the degrees of freedom enabled by the application, while the consequence graph states how those were made use of, by a particular runtime on particular hardware.
   1.250 +Here we describe the visual representations of a concrete UCC and an SCG. The UCC conveys the scheduling related structure of the application code, to provide understanding of what schedulings are possible. The SCG focuses on how the constraints in the application interact with the runtime and hardware. It displays the cause-and-effect cascades  and links each node in a cascade back to application code, runtime implementation, or hardware. 
   1.251 +
   1.252 +%The Unit \& Constraint Collection, or UCC, elements are the application's units of work  and the constraints on scheduling them. Constraints can be explicitly stated in the code, or  implied by language constructs. They limit the choices available to the runtime. We use the  general term ``constraint'' instead of the specific term ``dependency'' because dependency only covers  one pattern: this unit before that one. Meanwhile  constraints are general,  such as e.g. a mutual exclusion on scheduling a group of units: any order, but only one at a time.
   1.253 +
   1.254 +%We refer to the second visualization as a scheduling consequence graph (SCG), or just consequence graph, which is explained in \S \ref{sec:SCGExpl}. It depicts where the runtime assigned each of the units, how long the units executed, the sources of overhead, and the changes in constraint-state that triggered each runtime behavior. 
   1.255 +
   1.256 +%In short, the UCC states the degrees of freedom enabled by the application, while the consequence graph states how those were made use of, by a particular runtime on particular hardware.
   1.257  
   1.258  \subsubsection{UCC visualization} \label{sec:UCCExpl}
   1.259  
   1.260 @@ -192,7 +260,7 @@
   1.261  \subsubsection{SCG visualization}  \label{sec:SCGExpl}
   1.262  The Scheduling Consequence Graph is the main visualization used to detect chains of causal interactions between elements of the system.  As the  example from the introduction indicated, a unit may end, which sends a signal to the runtime to update the state of the unit and the state of the hardware resources it occupied. This  causes the runtime to choose a different unit to own that hardware and sends the meta-information for that unit to the hardware. This in turn triggers communication, to send the data consumed by the unit to the hardware. Then the work of the new unit takes place there.
   1.263  
   1.264 -Any one of these interactions could be individually abnormal, and an unexpected source of performance loss. The SCG allows selectively turning on visualization of each kind of causal interaction, until the culprit for the unexpected slowdown is seen.  
   1.265 +Any one of these interactions could be individually abnormal, and an unexpected source of performance loss. The SCG allows selectively turning on visualization of each kind of cause-and-effect, until the culprit for the unexpected slowdown is seen.  
   1.266  \begin{figure}[ht]
   1.267    \centering
   1.268    \includegraphics[width = 2in, height = 1.8in]{../figures/SCG_stylized_for_expl.pdf}
   1.269 @@ -201,23 +269,62 @@
   1.270  \end{figure}
   1.271  
   1.272  Fig. \ref{fig:SCG_expl} shows a consequence graph, stylized for purposes of explanation. 
   1.273 -It is composed of a number of columns, one for each core. A column represents time on the core,  increasing as one goes down, measured in clock cycles. It is broken into blocks, each representing the time accounted to one work-unit. Each block is further divided into regions, each a different color, which indicates the kind of activity the core was engaged in during that  region's time-span.
   1.274 -
   1.275 -The application code executed within a block is linked to the block. In our tool, the block is labelled with a unique unitID. This ID  is then linked to the code executed within that unit. In this way, the code of any block can be looked up, along with the parallelism constructs that mark the start and end of the block.
   1.276 -
   1.277 -The kinds of activities within a block are defined by the computation model that underlies the visualization. The first kind of activity is the actual work, plus  waiting for  cache misses. It is represented by a blue-to-red region where the color indicates  intensity of cache misses, with pure red representing at or above the maximum misses per instruction, and pure blue the minimum (the max and min are set in the tool that generates the visualization).
   1.278 -
   1.279 -  The second kind of activity is runtime overhead, represented by a gray region. This is the overhead spent on that particular work-unit. When desired by the user, it is further broken into pieces representing activities inside the runtime. The options include time spent on: constraints, when determining readiness of a work-unit; deciding which ready unit to assign to which hardware; and time spent switching from virtual processor, to the runtime, and back. In this paper, we show all runtime overhead lumped together, however in other circumstances a breakdown can be key to seeing where unexpected slowdown is taking place. 
   1.280 -
   1.281 -The other type of visual feature seen in Fig. \ref{fig:SCG_expl} is lines. Each represents a construct that influenced scheduling, where the color indicates what kind of construct.   A line represents two things: a constraint, whose satisfaction made the lower unit ready, and a decision by the runtime to start the lower unit on that core. 
   1.282 -
   1.283 -In general, lines may also be drawn that represent other kinds of interactions, which affect core usage. For example,  our runtime implementation only allows one core at a time to access shared  scheduling state. Visualization of this can be turned on, as additional lines linking the gray runtime regions of blocks (visualization of such interactions is turned off in this paper for simplicity). 
   1.284 -
   1.285 -Two work-unit blocks that appear in sequence and have no lines drawn between them often have a causal dependency, due to the semantics of the base language (visualization of these causalities is also turned off, but can be inferred via the link to the code).
   1.286 +Each column represents time on one core, increasing as one goes down, measured in clock cycles. Each box represents a unit, the time accounted to it. The categories on time are represented inside a box as regions, each a different color. The color indicates the kind of activity the core was engaged in during that region's time-span.
   1.287 +
   1.288 +A box is labelled with a unique unit ID that links to the code executed within that unit, as well as the parameters passed to it. With this the code of any box can be looked up, along with the parallelism constructs that mark the start and end of the unit.
   1.289 +
   1.290 +The first kind of activity is the actual work, combined with waiting for  cache misses. It is represented by a blue-to-red region where the color indicates  intensity of cache misses, with pure red representing at or above the maximum misses per instruction, and pure blue the minimum (the max and min are set in the tool that generates the visualization).
   1.291 +
   1.292 +  The second kind of activity is runtime overhead, represented by a gray region. This is the overhead spent on that particular work-unit. When desired by the user, it is further broken into pieces representing activities inside the runtime. The options include time spent on: constraints, when determining readiness of a work-unit; deciding which ready unit to assign to which hardware; and time spent switching from virtual processor, to the runtime, and back. 
   1.293 +
   1.294 +This paper is an introductory tutorial, so we simplify by setting the tool to show all runtime overhead lumped together. however in other circumstances a breakdown can be key to seeing where unexpected slowdown is taking place. 
   1.295 +
   1.296 +The other type of visual feature seen in Fig. \ref{fig:SCG_expl} is lines. Each represents a cause-and-effect, where the color indicates what kind of causality. In this paper, we set the tool to only show parallelism construct cause and effect.  Hence, a line represents two things: a constraint, whose satisfaction made the lower unit ready, and a decision by the runtime to start the lower unit on that core. The color, here, corresponds to the kind of construct.
   1.297 +
   1.298 +For more advanced tuning, lines can be turned on that represent the other kinds of cause-and-effect. For example, our SSR runtime implementation only allows one core at a time to access shared  scheduling state. Visualization can be turned on that draws a separate region for the time spent waiting to acquire the lock, along with additional lines that link lock-acquire regions of the box that release the lock to the region of the box that acquired it. 
   1.299 +
   1.300 +Two work-unit blocks that appear in sequence and that have no lines drawn between them often have a causal dependency, due to the sequential semantics of the base language.  For this tutorial, visualization of these causalities is also turned off, but can be inferred via the link to the code.
   1.301  
   1.302  Note that  many different orderings can be validly chosen by the runtime. The scheduler choices that are valid is determined by three kinds of constraints: the application code constraints, hardware constraints, and runtime implementation imposed constraints.
   1.303  
   1.304 -The visual features allow the user to see at a glance the total  execution time (height), idle cores during the run (empty space), cache behavior (color of work regions), degree of overhead (size of gray regions), and which units constrained which other units (lines). All consequence graphs in this paper are at the same scale, so they can be compared directly.
   1.305 +The visual features allow the user to see at a glance the total  execution time (height), idle cores during the run (empty space), cache behavior (color of work regions), degree of overhead (size of gray regions), and which units constrained which other units (lines). Note that all consequence graphs in this paper are at the same scale, so they can be compared directly.
   1.306 +
   1.307 +
   1.308 +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   1.309 +\section{Illustration of Tuning Scheduling Decisions}
   1.310 +\label{sec:casestudy}
   1.311 +
   1.312 +In this section we teach how to use the visualization to generate hypotheses about what to change in the code, in order to manage the cascades of cause-and-effect. 
   1.313 +
   1.314 +Because this is a tutorial, we create a contrived example  that brings out particular features of the tool, but still uses a real application running on real hardware.  The example uses features of SSR to explicitly assign units to cores, in the application source code.  This isn't the normal way that tuning would take place, but it allows us to control the activity to illustrate the concepts in the order most conducive to learning.
   1.315 +
   1.316 +We start  by describing the hardware being run on and structure of the program being tuned, and follow by describing the parallel language used and instrumented. 
   1.317 +
   1.318 + A sequence of visualizations follow. In each,  we point out how the  performance loss is identified, and which visual features suggest the hypothesis of the cause of the loss.  We show how the visual features direct the user to the specific sections of code that need to be changed, and how the model suggests what changes to try. 
   1.319 +
   1.320 +\subsection{Setup: Hardware and Application}
   1.321 +
   1.322 +We run our example on a machine with 4 sockets by 10 cores each, for a total of 40 physical cores. They are Intel WestmereEx cores running at 3.0GHz, with TurboBoost turned off for reproducability.
   1.323 +
   1.324 +For this tutorial, we chose a program that the reader knows well, matrix multiply.  This allows concentration on the tool without distraction about the application. 
   1.325 +
   1.326 +The application is structured as follows: it  creates one entity to divide the work into a number of pieces and  creates another entity for each piece of work. How many pieces is determined by the combination of a tuning parameter in the code together with the number of cores. The work is distributed across the cores  in a round-robin fashion, unless otherwise specified.
   1.327 +
   1.328 +The application also creates an entity that manages the partial-results. Each piece of work sends its contribution, which is accumulated into the overall  result. The entity that divides waits for the entity that accumulates to signal completion then the language runtime shuts down.
   1.329 +
   1.330 +\subsection{Programming Model}
   1.331 +We chose a simple language that was convenient to instrument. It is inspired by pi-calculus, and called Synchronous Send-Receive (SSR).  It implements  rendez-vous style send and receive operations made between virtual processors (VPs), where a VP is similar to a software thread. The example application uses the commands for creating and destroying VPs, two kinds of send-receive paired operations, a parallel singleton, and scheduling control constructs. 
   1.332 +
   1.333 +The first kind of send-receive pair is precise about sender and receiver. Called \emph{send\_from\_to}, it specifies both sender and receiver VPs, and is used by the results VP  to tell the divider VP that the work is complete. The second kind, \emph{send\_of\_type\_to}, specifies only a specific receiver, leaving the sender anonymous. It has increased flexibility while maintaining some control over scope. The worker VPs use this to send their partial result to the results VP. 
   1.334 +
   1.335 +The application uses the \emph{singleton} construct to reduce the amount of work done by the (sequential) divider VP. The construct designates a piece of code as to be executed only once, even though it is invoked by multiple VPs. It is employed to shift the work of copying matrix fragments out of the divider and over to the worker-pieces. The first worker-piece to use a given input-matrix fragment performs the copy, which spreads the copying across the cores.
   1.336 +
   1.337 +For this tutorial, we control the scheduling behaviors, in order to concisely illustrate the tool. This is done with language constructs that force which core a virtual processor is assigned to. 
   1.338 +
   1.339 +A note on terminology: We often use the term  ``work-unit'',  which we define precisely, instead of ``task'', which has acquired multiple  meanings in the literature. Work-unit  is defined as the trace-segment performed on a core, between two successive scheduling events, plus the set of datums consumed by that trace segment. The word task often maps well onto this  definition, and we use both words, but mean the precise work-unit definition when we say task.
   1.340 +
   1.341 +%<snip>
   1.342 +
   1.343  \begin{figure*}[t!]
   1.344    \begin{minipage}[b]{0.25\textwidth}
   1.345          \hfill\subfloat[35.8 Gcycles\\Original]
   1.346 @@ -269,7 +376,7 @@
   1.347  \subsection{Walk-through}
   1.348  \label{subsec:walk-through}
   1.349  
   1.350 -We wish to show the visualizations in a simple  way, to enhance understanding. Hence, this walk through uses a slightly contrived example in which the application explicitly controls where each unit of work is assigned. As a result, the causal interactions of interest are all constraints stated by language constructs. We don't show causalities internal to the runtime system or hardware, although the tool is capable of turning on display of those. We chose to do so to simplify this introduction to the use of the visualizations.  
   1.351 +We wish to show the visualizations in a simple  way, to enhance understanding. Hence, this walk through uses a contrived example in which the application explicitly controls where each unit of work is assigned. As a result, the causal interactions of interest are all constraints stated by language constructs. We don't show causalities internal to the runtime system or hardware, although the tool is capable of turning on display of those. We chose to do so to simplify this tutorial on the use of the visualizations.  
   1.352  
   1.353  Fig. \ref{story} displays all of the scheduling consequence graphs generated during our tuning session. They all use the same scale, for direct comparison. All have 40 columns, one for each core, and relative height indicates relative execution time. The lines in red, orange, and green represent application-code constructs. Red is creation of a virtual processor, green is the many-to-one \texttt{send\_of\_type\_to}, and orange is the singleton construct. For better visibility, only constraints that cross cores are turned on.
   1.354  
   1.355 @@ -300,24 +407,28 @@
   1.356  
   1.357  \subsubsection{Holes in the core usage}\label{subsec:holes}
   1.358  
   1.359 -In Fig \ref{story:e}, the true value of the SCG starts to appear. In it, ``holes'' are noticeable. Inspecting these holes closer, we can see that the stalled blocks are at the ends of orange lines. This tells us definitively that they were waiting upon the completion of a singleton. Other tools may have indicated that singleton constructs have a mild time spent blocked, but they wouldn't show how a single blocked singleton had a  chain-reaction effect, holding up creation, then in turn causing many cores to sit idle. 
   1.360 -
   1.361 - Zooming in on the singletons and tracing out the state inside the runtime shows that the empty space  is a runtime implementation issue. This  analysis required all the forms of information provided by the SCG.  such as  the knowledge of precisely which units came before and after the one blocked by the singleton, in combination with  the fact that it is a singleton construct blocking.  
   1.362 +In Fig. \ref{story:e}, the true value of the SCG starts to appear. In it, ``holes'' are noticeable. Inspecting these holes closer, we can see that the stalled blocks are at the ends of orange lines. This tells us definitively that they were waiting upon the completion of a singleton. Other tools may have indicated that singleton constructs have a mild time spent blocked, but they wouldn't show how a single blocked singleton had a  chain-reaction effect, holding up creation, then in turn causing many cores to sit idle. 
   1.363 +
   1.364 + Zooming in on the singletons and tracing out the state inside the runtime shows that the empty space  is a runtime implementation issue. This  analysis required all the forms of information provided by the SCG,  such as  the knowledge of precisely which units came before and after the one blocked by the singleton, in combination with  the fact that it is a singleton construct blocking.  
   1.365  
   1.366  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   1.367 -\section{The Model Behind the Visualization}
   1.368 +\section{Advanced Application of UCC and SCG}
   1.369  \label{sec:theory}
   1.370 -The value of the visualizations comes from the causality linkages. Being able to trace causes from idle cores back through hardware, runtime implementation, and scheduling choices, provides the added insight that augments the other tools. This linkage tracing is in turn made possible by the model of the structure of computation.    In effect, the tool directly visualizes the model.
   1.371 -
   1.372 -As seen, the model has two parts, a \emph{Unit \&\ Constraint Collection (UCC)}, and a \emph{Scheduling Consequence Graph} (SCG or just consequence graph).  The UCC depicts the scheduling  choices  the application allows, and so shows what the programmer has control over. The consequence graph says which of those were actually taken during the run and the causal linkage of consequences within the hardware, runtime, and succeeding choices.
   1.373 -
   1.374 -In this section, we give a more precise description of UCC, then consequence graph.
   1.375 -However, this paper focuses on their application to performance tuning, so we abbreviate here and give a formal definition of the full model in a different paper.
   1.376 +%The value of the visualizations comes from the causality linkages. Being able to trace causes from idle cores back through hardware, runtime implementation, and scheduling choices, provides the added insight that augments the other tools. This linkage tracing is in turn made possible by the model of the structure of computation.    In effect, the tool directly visualizes the model.
   1.377 +
   1.378 +As seen, the model has two parts, a \emph{Unit \&\ Constraint Collection (UCC)}, and a \emph{Scheduling Consequence Graph} (SCG or just consequence graph).  The UCC depicts the permissible scheduling  choices, and so shows what the programmer has control over. The consequence graph says which of those were actually taken during the run and the causal linkage of consequences within the hardware, runtime, and onto down the line scheduling  choices.
   1.379 +
   1.380 +In this section, we convey how the UCC and SCG are applied to more advanced situations. We give a formal definition of the full model in a different paper.
   1.381 +
   1.382  \subsection{Unit \& Constraint Collection}
   1.383  \label{sec:UCC}
   1.384 -A fully specified UCC contains the units of work that get scheduled during a run, and the constraints placed on scheduling those units.  In the simple case, all units and all constraints are fully specified in the application. However, many classes of application exist, and two degrees of freedom  determine how much of the UCC is actually defined in the application vs the input data, or even in the runtime.
   1.385 -
   1.386 -An example of a simple application that has all units and all constraints fixed in the source  is matrix multiply with fixed size matrices.  For other applications, the shape of the UCC is only partially defined by the application code.  Take the matrix multiply used in Section \ref{sec:casestudy}, where an input parameter determined the number of units created. There, the UCC was different for each parameter value. At the extreme, would be parallel search  with load-driven pruning. Pruning the search space means that  the units themselves are a function  of both the input data \emph{and} the previous pruning decisions made by the runtime.
   1.387 +It turns out that for many applications, not all units can be determined just from the source code.  Instead, additional factors come into the picture, namely parameters, input data and runtime choices.  In fact, the constraints also might be determined by parameters, input data and/or runtime choices.
   1.388 +
   1.389 +For example, for matrix multiply with sizes of matrix fixed in the application code and know number of cores, all units and constraints can be determined just from the application.  But if the code instead accepts the matrix size as an input parameter, then the units are no longer known just from the source code.
   1.390 +
   1.391 +We call a UCC that contains all the units, with all constraints on them, a fully concrete UCC.  Meanwhile, a UCC that requires additional input before becoming fully concrete, we call an abstract UCC.
   1.392 +
   1.393 +A UCC can be abstract in the units, or abstract in the constraints, or both. This means that there is more than just one concrete UCC for an application source that defines an abstract UCC.
   1.394  
   1.395  \begin{figure}[ht]
   1.396    \centering
   1.397 @@ -326,78 +437,89 @@
   1.398    \label{fig:UCC_example}
   1.399  \end{figure}
   1.400  
   1.401 -    We call a fully specified UCC a \emph{concrete} UCC.  Every run of an application eventually winds up defining a concrete UCC. The example in   Fig. \ref{fig:UCC_example} was produced for the performance tuning. Every unit that was scheduled in the SCG appears in it,  along with  the application-defined constraints on scheduling them.  The division parameter  determined the units. Hence, the application alone does not specify the concrete UCC, because the units remain unknown until parameter values are given. 
   1.402 -
   1.403 -In general, the amount of UCC made concrete by the application  falls into a two-dimensional grid. One dimension covers the units, the other the constraints, as shown in 
   1.404 -Fig. \ref{fig:UCC_Concreteness}.  The two axes each have  four kinds of  information that has to be added in  order to determine the units and constraints,  in the final concrete UCC. 
   1.405 -
   1.406 -The UCC may change at multiple points in an application's lifecycle. The position  a UCC lands on the grid indicates how far it is from being fully concrete.  The horizontal   position indicates what inputs are still needed to determine the units, and vertical the constraints.  0 indicates that the units (constraints) are fully determined by the application code alone; 1 means parameter values also must be known; 2 means input data values also play a role, and 3 means runtime decisions play a role in determining the units (constraints).
   1.407 -
   1.408 -The concept of UCC, and  its concreteness can provide value in classifying applications and algorithms and predicting what types of scheduling approach and hardware they will perform best on. The concepts also help in understanding what optimizations do to code and what changes may improve performance on given hardware plus runtime. 
   1.409 +% We call a fully specified UCC a \emph{concrete} UCC.  
   1.410 +
   1.411 +Every run of an application eventually winds up defining a concrete UCC. For  example    Fig. \ref{fig:UCC_example} was produced during the performance tuning done in \S\ref{sec:story}. Every unit that was scheduled in the SCG appears in it,  along with  the application-defined constraints on scheduling them. However, it was only after knowing the division parameter  that the units could be determined.  The added information turned the original abstract UCC into a fully concrete one. 
   1.412 +
   1.413 +%A fully specified UCC contains all the units of work that get scheduled during a run, and the constraints placed on scheduling those units.  In the simple case, all units and all constraints are fully specified in the application. However, many classes of application exist, and two degrees of freedom  determine how much of the UCC is actually defined in the application vs by the input data, or even in the runtime.
   1.414 +
   1.415 +
   1.416 +
   1.417 +%An example of a simple application that has all units and all constraints fixed in the source  is matrix multiply with fixed size matrices.  For other applications, the shape of the UCC is only partially defined by the application code.  Take the matrix multiply used in Section \ref{sec:casestudy}, where an input parameter determined the number of units created. There, the UCC was different for each parameter value. At the extreme, would be parallel search  with load-driven pruning. Pruning the search space means that  the units themselves are a function  of both the input data \emph{and} the previous pruning decisions made by the runtime.
   1.418 +
   1.419 +In general, the amount of UCC made concrete by the application  falls into a two-dimensional grid. One dimension covers the units, the other the constraints, as shown in Fig. \ref{fig:UCC_Concreteness}.  In each axis there are four kinds of  information that determine the units and constraints of a final concrete UCC: 0) application code, 1) parameters, 2) input data, and 3) runtime/scheduling choices. 
   1.420  
   1.421  \begin{figure}[ht]
   1.422    \centering
   1.423    \includegraphics[width = 2in, height = 1.8in]{../figures/UCC_concreteness_grid.pdf}
   1.424 -  \caption{Abstract representation of the amount  of concreteness of a UCC. The letters A, B, C, D stand for UCCs described in the text.}
   1.425 +  \caption{Grid representing the kinds of inputs required in order to form a fully concrete UCC.  Most UCCs fit into a single square. The letters A, B, C, D show where UCCs described in the text land.}
   1.426    \label{fig:UCC_Concreteness}
   1.427  \end{figure}
   1.428  
   1.429 -In the concreteness grid, the closer an application-derived UCC is to the origin, the less additional information is needed to obtain a  concrete UCC descendant of it. For example, the UCC labeled A in the figure is fully concrete just from the source code alone. It represents, for example, matrix multiply with fixed size matrices and fixed division. The UCC labeled B requires the input data plus parameters to be specified before its units are concrete, but just parameters to make its constraints fully concrete. Ray-tracing with bounce depth specified as a parameter may be like this. The UCC labeled C only has variability in its constraints, which require input data. An example would be H.264 motion vectors.
   1.430 -But even the least concrete UCC, D, at the end of the diagonal, generates a concrete descendant UCC while a run of the application unfolds.
   1.431 +The position   a UCC lands on the grid indicates how far it is from being fully concrete.  The horizontal   position indicates what inputs are still needed to determine the units, and vertical the constraints.  
   1.432 +
   1.433 +The UCC labeled A in the figure is fully concrete from the source code alone. An example would be matrix multiply with fixed size matrices divided into fixed pieces. The UCC labeled B requires the input data plus parameters to be specified before its units are concrete, but only needs parameters to make its constraints fully concrete. Ray-tracing with bounce depth specified as a parameter may be an example of this. The UCC labeled C only has variability in its constraints, which require input data. An example would be H.264 motion vectors, which has dependencies determined by the values of pixels in a video frame sequence.
   1.434 +The least concrete UCC, D, is at the end of the diagonal.  However, even it generates a concrete descendant UCC as the run of the application unfolds.
   1.435 +
   1.436 +%In the concreteness grid, the closer an application-derived UCC is to the origin, the less additional information is needed to obtain a  concrete UCC descendant of it. The UCC labeled A in the figure is fully concrete just from the source code alone. It represents, for example, matrix multiply with fixed size matrices and fixed division. The UCC labeled B requires the input data plus parameters to be specified before its units are concrete, but only needs parameters to make its constraints fully concrete. Ray-tracing with bounce depth specified as a parameter may be an example of this. The UCC labeled C only has variability in its constraints, which require input data. An example would be H.264 motion vectors.
   1.437 +
   1.438 +
   1.439 +Various things can modify a UCC, such as compiler toolchain, auto-tuning, and the runtime system. For example, polyhedral toolchains derive the constraints hidden within sequential code, then calculate how to divide the work into  units and schedule them onto given  hardware. The result is a new UCC that has parallelism where the original did not. 
   1.440 +
   1.441 +%An application's lifecycle spans from editing code, to the toolchain during build, all the way through  the run. Its representation  may change at each stage of life, with corresponding changes to the UCC.  For example, specialization may perform a static scheduling, which fixes the units, moving the UCC towards the origin. 
   1.442 +
   1.443 +Alternatively, the toolchain may inject manipulator code, which the runtime later uses to divide units during the run. The result is a more abstract UCC because the manipulator code gives the runtime control to define the units, moving the UCC further from the origin.
   1.444 +
   1.445 +
   1.446 +
   1.447 +The UCC, and  its concreteness, can be used to classify applications, and predict what  scheduling approach and hardware the application will perform best on. The concepts also help to understand why a given parallelism-related optimization improves performance, and to predict what changes may improve performance. 
   1.448 +
   1.449 +
   1.450 +The degree of abstractness of a UCC indicates what is inside the application programmer's control vs determined by input data or runtime implementation choices. In particular, the application-derived UCC  shows what can be done statically: the further from the origin that UCC lies, the less scheduling can be done statically in the toolchain.
   1.451 +
   1.452   
   1.453 -Bear in mind that  even a fully concrete UCC still has degrees of freedom when deciding which units to run on which hardware and in what order of execution. Those decisions determine interactions within the hardware, to yield the communication patterns and consequent performance  during the run,  visualized by the SCG. 
   1.454 -
   1.455 -
   1.456 -
   1.457 -As noted, an application has a lifecycle, spanning from editing code all the way through  the run, and its representation  may change at the different stages of life, with corresponding changes to the UCC.
   1.458 - For example, specialization may perform a static scheduling, which fixes the units, moving the UCC towards the origin. Alternatively, the toolchain may inject manipulator code for the runtime to use, which lets it divide units during the run when it needs more units. The injection of manipulator code makes the UCC less concrete, moving it further from the origin.
   1.459 -
   1.460 -The progression of UCCs has value in performance tuning  because it indicates what is inside the application programmer's control vs under control of each tool in the toolchain or  the runtime. For example, the original application-derived UCC  shows what can be done statically: the further out on the diagonal that UCC is, the less scheduling can be done statically in the toolchain.
   1.461 -
   1.462 -In this paper, we do not suggest how to represent UCCs far out on the diagonal. One of those actually indicates a multi-verse of concrete-UCCs. Which of them materializes  depends on the data and what the scheduler decides. We only represent the concrete UCC that materializes during a run and leave the question of representing less concrete ones to future work.
   1.463 -
   1.464 -\subsection{Scheduling Consequence Graph}
   1.465 -Whereas the UCC has value in understanding the structure of the application,  the second part of the model shows how that structure  was made use of during a run. We called it the Scheduling Consequence Graph because it links scheduling decisions to their performance consequences. It does so by indicating distinct activities in the fundamental logical structures, and linking them causally to each other. As a result it provides a link from each instance of lost performance, back through each element that contributed to causing the loss, as illustrated in Section \ref{subsec:holes}.
   1.466 -
   1.467 -Contrary to the UCC, the consequence graph shows the behavior resulting from scheduling decisions actually \emph{made},   from among those \emph{possible.} The UCC shows just the possibilities. Hence, a consequence graph shows \emph{one} of the possible choice-sets allowed by a concrete UCC.
   1.468 -
   1.469 -A consequence graph accounts for each bit of core time, arranged into boxes and arcs, with boxes further divided into regions. A box  represents all core time assigned to one work unit. Each region inside represents a segment of time that the core was engaged in a specific type of activity.  An arc links  regions (or boxes)  and represents a causality of some kind.
   1.470 -
   1.471 -There is one kind of region for each kind of activity that can source or sink a causal link, and a region may source or sink multiple causalities. Each type of causal link is represented by a corresponding type of arc.
   1.472 -
   1.473 -The core activities/\ region-types are: doing application work, waiting idly for communication of work data, managing constraints, deciding assignment of work onto cores, and doing internal runtime activities. The runtime region has sub-activities, but we do not detail them here due to space. However, some will be stated in Section \ref{sec:Implementation} when we talk about instrumenting a runtime.
   1.474 -
   1.475 -Arcs are gathered into groups according to the nature of the causality they represent.  The kinds of causal links are: satisfaction of a sequential dependency in the base language; satisfaction of a parallel constraint  (i.e., one unit did something to satisfy a constraint on the other, causing it to be free to be scheduled); a causal link internal to the runtime (for example, the runtime on one core releasing a shared lock, causing the other core to acquire it); and causal links in the hardware (for example, one work-unit finishes on a core, causing another work-unit to start there, modulo a choice by the runtime).
   1.476 -
   1.477 -We will now expand on each of those kinds of causal link.
   1.478 -
   1.479 -\paragraph{Constraint  causal link} Two entire boxes (units) are linked this way when   action by one unit contributes to satisfaction of a constraint blocking the other unit. This includes sequential dependencies from the base language (which are noted in the tool but normally not displayed).
   1.480 -
   1.481 -Sequential dependencies may add superfluous constraints that  eliminate some otherwise allowed choices in the UCC. An example would be a \texttt{for} loop that creates work-units -- no parallelism constructs cause the creations to be done in sequence, but the base C language sequentializes it nonetheless. 
   1.482 -
   1.483 -\paragraph{Runtime internal causal link} Runtime implementation details may introduce ``extra" causalities between units. For example, the runtime we instrumented for this paper runs separately on each core and relies upon a global lock for accessing shared runtime information. This lock introduces a causal relationship  when the runtime on one core is attempting to process one unit, but must wait for the runtime on a different core to finish with its unit.
   1.484 -
   1.485 - Normally, these are not displayed explicitly, due to clutter, but can be turned on when needed, for instance to determine the cause of a particular pattern of core usage.
   1.486 -
   1.487 -\paragraph{Hardware causal link} The physical fact that a given resource can only be used by one work-unit at a time introduces hardware causalities. When multiple units are free to execute, but all cores are busy, then completion of a unit  on one core causes (in part) the next ready unit to run on that core. 
   1.488 -
   1.489 -These are also not normally displayed, due to clutter, and not all hardware dependencies are directly measured. Future work will focus on using the performance counters and other instrumentation to add more information about communication paths taken as a consequence of the scheduling decisions made. It will start with the current linkage of application-code to runtime decisions, and add consequent usage of communication hardware. This gives an end-to-end linkage between runtime choices and caused behavior on the hardware. 
   1.490 -
   1.491 -
   1.492 -\subsection{Breadth of Applicability of UCC and SCG}
   1.493 -
   1.494 -The question arises of how widely applicable the model is. Especially for heterogeneous machines, such as ones that have multiple levels of runtime, which implies multiple levels of scheduler. What do the UCC and SCG look like in this case?
   1.495 -
   1.496 -
   1.497 -
   1.498 -
   1.499 -To answer this, let's explore a specific case of a cluster in one rack. The rack has a back-plane that boards plug into, while a board has its own memory and four sockets, each having a chip with four cores. Correspondingly it has separate physical networks:  a back-plane network connecting the boards; an inter-socket network on each board; and a  cache hierarchy connecting cores in a socket.
   1.500 +%Bear in mind that  even for a fully concrete UCC, it is the runtime that decides which units to run on which hardware and in what order of execution. Those decisions determine interactions within the hardware, in the form of communication patterns and consequent performance  during the run. All visualized by the SCG. 
   1.501 +
   1.502 +
   1.503 +
   1.504 +In this paper, we do not suggest how to represent abstract UCCs.  One actually indicates a multi-verse of concrete-UCCs. Which of them materializes  depends on the data and what the scheduler decides. We only represent the concrete UCC that materializes during a run and leave the question of representing abstract ones to future work.
   1.505 +
   1.506 +\subsection{Applying UCC and SCG to hierarchical hardware}
   1.507 +%Whereas the UCC has value in understanding the structure of the application,  the second part of the model shows how that structure  was made use of during a run. We called it the Scheduling Consequence Graph because it links scheduling decisions to their performance consequences. It does so by indicating distinct activities in the fundamental logical structures, and linking them causally to each other. As a result it provides a link from each instance of lost performance, back through each element that contributed to causing the loss, as illustrated in Section \ref{subsec:holes}.
   1.508 +
   1.509 +
   1.510 +%A consequence graph accounts for each bit of core time, arranged into boxes and arcs, with boxes further divided into regions. A box  represents all core time assigned to one work unit. Each region inside represents a segment of time that the core was engaged in a specific type of activity.  An arc links  regions (or boxes)  and represents a causality of some kind.
   1.511 +
   1.512 +%There is one kind of region for each kind of activity that can source or sink a causal link, and a region may source or sink multiple causalities. Each type of causal link is represented by a corresponding type of arc.
   1.513 +
   1.514 +%The core activities/\ region-types are: doing application work, waiting idly for communication of work data, managing constraints, deciding assignment of work onto cores, and doing internal runtime activities. The runtime region has sub-activities, but we do not detail them here due to space. However, some will be stated in Section \ref{sec:Implementation} when we talk about instrumenting a runtime.
   1.515 +
   1.516 +%Arcs are gathered into groups according to the nature of the causality they represent.  The kinds of causal links are: satisfaction of a sequential dependency in the base language; satisfaction of a parallel constraint  (i.e., one unit did something to satisfy a constraint on the other, causing it to be free to be scheduled); a causal link internal to the runtime (for example, the runtime on one core releasing a shared lock, causing the other core to acquire it); and causal links in the hardware (for example, one work-unit finishes on a core, causing another work-unit to start there, modulo a choice by the runtime).
   1.517 +
   1.518 +%We will now expand on each of those kinds of causal link.
   1.519 +
   1.520 +%\paragraph{Constraint  causal link} Two entire boxes (units) are linked this way when   action by one unit contributes to satisfaction of a constraint blocking the other unit. This includes sequential dependencies from the base language (which are noted in the tool but normally not displayed).
   1.521 +
   1.522 +%Sequential dependencies may add superfluous constraints that  eliminate some otherwise allowed choices in the UCC. An example would be a sequential \texttt{for} loop that creates work-units -- no parallelism constructs cause the creations to be done in sequence, but the base C language sequentializes it nonetheless. 
   1.523 +
   1.524 +%\paragraph{Runtime internal causal link} Runtime implementation details may introduce ``extra" causalities between units. For example, the runtime we instrumented for this paper runs separately on each core and relies upon a global lock for accessing shared runtime information. This lock introduces a causal relationship  when the runtime on one core is attempting to process one unit, but must wait for the runtime on a different core to finish with its unit.
   1.525 +
   1.526 +% Normally, these are not displayed explicitly, due to clutter, but can be turned on when needed, for instance to determine the cause of a particular pattern of core usage.
   1.527 +
   1.528 +%\paragraph{Hardware causal link} The physical fact that a given resource can only be used by one work-unit at a time introduces hardware causalities. When multiple units are free to execute, but all cores are busy, then completion of a unit  on one core causes (in part) the next ready unit to run on that core. 
   1.529 +
   1.530 +%These are also not normally displayed, due to clutter, and not all hardware dependencies are directly measured. Future work will focus on using the performance counters and other instrumentation to add more information about communication paths taken as a consequence of the scheduling decisions made. It will start with the current linkage of application-code to runtime decisions, and add consequent usage of communication hardware. This gives an end-to-end linkage between runtime choices and caused behavior on the hardware. 
   1.531 +
   1.532 +
   1.533 +%\subsection{Breadth of Applicability of UCC and SCG}
   1.534 +
   1.535 +The question arises of how to apply the UCC and SCG to heterogeneous machines that have multiple levels of runtime, and thus multiple levels of scheduler. To answer that, let's explore a specific machine. In this case, a cluster in one rack. The rack has a back-plane that boards plug into, while a board has its own memory and four sockets, each socket holds a chip with four cores. This system has corresponding physically separate networks:  a back-plane network connecting the boards; an inter-socket network on each board; and a  cache hierarchy connecting cores within a socket.
   1.536  
   1.537  A runtime is provided for each network, to distribute work among the networked components. The top runtime divides work among the boards, then a board's runtime divides work among the sockets, and a socket's  runtime divides work among the cores.  
   1.538  
   1.539  To a given runtime, each runtime below it is treated as an indivisible  machine. So the backplane runtime treats each board as a complete machine, creates work for each board, and schedules it. To send work to a board, it  communicates with the runtime that manages that board, but without knowing the internal details of that runtime.  It just sees the runtime as an opaque ``machine" that it gives work to. Hence the top runtime schedules onto board-level runtimes, which each schedule onto socket-level runtimes, and so on.This multi-level nature should be reflected in the UCC and SCG. 
   1.540  
   1.541 -To facilitate this approach, the application  has been written in a language that provides a construct for work-unit division. For each construct instance, the toolchain inserts a corresponding manipulator, which enables each runtime to divide up the work it is given into smaller work-units. Examples include the DKU pattern \cite{DKUTechRep_09}, TBB's loop constructs \cite{LangTBBRef}, HPF's matrix division facilities \cite{LangHPF}, and so on. The effect of insertion of the manipulator on the \textit{application} UCC is to push it all the way to the right on the unit axis.  
   1.542 +To facilitate this approach, the application  has been written in a language that provides a construct for work-unit division. This construct provides a manipulator to the runtime, giving the runtime the ability to dynamically  divide up the work it is given into smaller work-units. Examples include the DKU pattern \cite{DKUTechRep_09}, TBB's loop constructs \cite{LangTBBRef}, HPF's matrix division facilities \cite{LangHPF}, and so on. Note that with such a manipulator, the UCC is maximally abstract on the unit axis.  
   1.543  
   1.544  \subsubsection{Levels of UCC}So what does the \textit{concrete} UCC produced during a run on this machine look like? Well,  scheduling decisions bound a work-unit, and the decisions take place inside a runtime. Hence the  units are defined relative to a runtime. Each runtime, then has its own concrete UCC for the units it schedules.
   1.545  \begin{figure}[ht]
   1.546 @@ -407,18 +529,16 @@
   1.547    \label{fig:UCC_Levels}
   1.548  \end{figure}
   1.549  
   1.550 -As seen in Fig. \ref{fig:UCC_Levels}, the concrete UCCs are related to each other in the same hierarchy as the runtimes. A single unit scheduled by one runtime has an entire concrete UCC in the level below. The unit is broken into sub-units, each separately scheduled by the lower-level runtime, which defines a new concrete UCC. Its units are those sub-units, and the constraints are the constraints amongst those. As the figure shows,  each unit in one UCC  has an entire lower-level UCC inside it, in a hierarchy matching the runtimes.
   1.551 -
   1.552 -\subsubsection{Levels of Consequence Graph}
   1.553 -What does this multi-level UCC mean for the SCG? Well, each level of scheduling activity has an associated scheduling consequence graph, hence each concrete UCC has a corresponding SCG. 
   1.554 -
   1.555 -But, in the  SCG we've seen, each column represents time on a core, so what does it mean in a higher-level SCG, for which a column represents a runtime instead of a core? This comes down to discovering the consequence of scheduling a given work unit onto  a particular runtime. When the goal is  improving performance, the consequence of interest is the time taken until completion of the work unit.
   1.556 -
   1.557 -Hence, the question becomes, when can the upper-level work unit be considered completed? When all the lower-level units of which it is comprised are completed, plus any additional work spent dividing up the work before and combining results after. Once the lower-level consequence graph is established, this  time can be easily determined: it is the critical path across the lower-level consequence graph plus the additional, non-overlapped, time spent creating the lower-level units and combining results into the format of the higher-level unit. The divide and recombine are not part of the  lower level UCC/SCG, because that is creation of them.
   1.558 +As seen in Fig. \ref{fig:UCC_Levels}, this means there are multiple levels of  concrete UCC, which  are related to each other in the same hierarchy as the runtimes. A single unit that is scheduled by one level of runtime has an entire concrete UCC in the level below. That unit is broken into sub-units, each of those is separately scheduled within the lower-level runtime. Hence, there's a whole lower-level UCC inside that unit, as the figure shows. The UCC hierarchy matches the  runtime hierarchy, which matches the hardware hierarchy.
   1.559 +
   1.560 +\subsubsection{Levels of SCG}
   1.561 +Each concrete UCC has a corresponding SCG. In the  SCG definition given above,  each column represents time on a core. But in a higher-level SCG, a column represents a runtime instead of a core. The consequence thus shifts to that of scheduling a given work unit onto  a particular runtime. Now, if the tuning goal is to  improve performance, then the consequence of interest is the time taken until the lower runtime completes the higher level work unit.
   1.562 +
   1.563 +The completion is when all the lower-level units are completed, plus any additional work spent dividing up the work before and combining results after. Once the lower-level consequence graph is established, this  time can be easily determined: it is the critical path across the lower-level consequence graph plus the additional, non-overlapped, time spent creating the lower-level units and combining results into the format of the higher-level unit. The divide and recombine are not part of the  lower level UCC/SCG, because the divide creates  the lower concrete UCC.
   1.564  
   1.565  \subsubsection{Applying the multi-level UCC and SCG}     
   1.566  
   1.567 -Let us apply this multi-level definition  to  the matrix multiply  from Section \ref{sec:casestudy}. We start at a level above the process that was created to execute the application. Here, a unit is the execution of an entire application, invoked via OS commands. The UCC at that level  treats each application execution as a unit. The SCG at that level treats the "runtime" region as the time spent by the OS creating a new process and running the loader and linker. The work time of a unit is the execution time of the application, from the first instruction until it exits.
   1.568 +Let us apply this multi-level definition  to  the matrix multiply  from Section \ref{sec:casestudy}. We start at a level above the process that was created to execute the application. Here, a unit is the execution of an entire application, invoked via OS commands. The UCC at that level  treats each application execution as a unit. The SCG at that level treats the ``runtime" region as the time spent by the OS creating a new process and running the loader and linker. The work time of a unit is the execution time of the application, from the first instruction until it exits.
   1.569  
   1.570  The application then contains code that breaks up the work, creates virtual processors to perform the work, and code that collects the results.  Breaking up the work is the calculations that determine the dimensions of each sub-matrix, and pairs up the sub-matrices. Each pair makes one work-unit, and has a virtual processor created to perform the multiply. These work-units comprise the second-level UCC, which is the UCC shown in \S\ref{sec:UCC}. The corresponding second-level SCGs are the ones shown during the walk-through.   
   1.571  
   1.572 @@ -596,7 +716,7 @@
   1.573  \vspace{-3pt}
   1.574  \begin{itemize}
   1.575  \setlength{\itemsep}{0em}
   1.576 -\item  it visualizes the essential factors influencing scheduling decisions and resulting causal chains, given sufficient language and runtime pre-requisites
   1.577 +\item  it visualizes the essential factors influencing scheduling decisions and resulting causal chains
   1.578  \item it has a filtering mechanism that shows only the most relevant groups of constraints and causal interactions, as desired
   1.579  \item it has flexibility to capture and represent any type of constraint
   1.580  
   1.581 @@ -610,7 +730,7 @@
   1.582  
   1.583  For message passing based software, many tools are available such as Paraver \cite{PerfToolParaver}, Vampir \cite{PerfToolVampir}, Paradign \cite{PerfToolParadyn}, and Tau \cite{PerfToolTau}. These give insight into which tasks communicate to which other tasks, which adds more causality information than available in the thread oriented tools.  
   1.584  
   1.585 -But a frequent complaint of users of these tools is that the cause of idleness is difficult to deduce.  One reason is that semantic information about messages is missing.  Little distinguishes them besides size, which also makes it difficult to filter the messages to prevent uninteresting ones from causing clutter. Although the tool we propose measures the same message interactions, plus additional sources of causal interactions, it has the ability to declutter based on the kind of causality, such as runtime-implementation internal causality, and also filter based on the semantic purpose of a given message.  The potential also exists to pick a particular idle period and display a trace of the causal chain that contributed to it.
   1.586 +But a frequent complaint of users of these tools is that the cause of idleness is difficult to deduce.  One reason is that semantic information about messages is missing.  Little distinguishes them besides size, which also makes it difficult to filter the messages to prevent uninteresting ones from causing clutter. The tool we propose measures the same message interactions, but adds additional sources of causal interactions. It also has the ability to declutter based on the kind of causality, such as runtime-implementation internal causality, and also filter based on the semantic purpose of a given message.  Once  a particular idle period is spotted, the tool can  display a trace of all the relevant scheduling causes that contributed to it.
   1.587  
   1.588  The HPCToolkit \cite{PerfBlameShifting12} uses blame-shifting to identify which constructs spend a  long time blocked. However, once identified, the user  could still use help in understanding  why  constructs block for so long, and why no other work fills in.
   1.589  
   1.590 @@ -654,11 +774,11 @@
   1.591  \section{Conclusion}
   1.592  \label{sec:conclusion}
   1.593  
   1.594 -We have shown how to apply a recently discovered  model of the structure of parallel computation to generate  visualizations of the sequence of interactions that cause  cores to sit idle during a run. Visualization features help identify suspect causal interactions and link them back to related application code. The approach relies only on information collected through instrumenting the language runtime, with no modification to the application.
   1.595 -
   1.596 -The essential causes for idleness are collected and available for display, including constraints due to application structure, language runtime implementation, and hardware features. The semantic information collected allows filtering   for the relevant types of constraints, to avoid overcharging the display.
   1.597 -
   1.598 -We demonstrated, with a case study, how to use the approach, which can provide a valuable addition to other performance tuning tools once tuning has shifted to scheduling related phenomena. The approach can reduce  guesswork, providing a  path to details in the application code where changes should be made.  
   1.599 +We have shown how to apply a recently discovered  theory of  parallel computation to generate  visualizations of causes and effects within the scheduling of work during a run. The theory defines visual features that go down inside the runtime implementation and link previously hidden causes there,  to related application code features. The approach doesn't modify the application, rather it instruments the runtime of the programming language.
   1.600 +
   1.601 +New details available for display include time spent inside meaningful  aspects of the runtime implementation,  how those cause effects within the scheduling activity, and how relevant hardware behaviors cause effects within the scheduling of work. The theory defines categories of activity, which can be used in filters   that de-clutter the display, to reveal the most relevant information.
   1.602 +
   1.603 +We gave a tutorial introduction to the approach. It can be used to enhance  tuning done in concert with other tools. The extra meaning provided by the new kinds of detail reduces  guesswork, elucidating all relevant levels of interaction that combine to cause performance loss. This enhances generating the correct hypothesis for the changes to application code and/or tools that improve performance.  
   1.604  
   1.605  %I'd like to avoid weaknesses of our approach, in the conclusion.. and this wasn't discussed much in the body.
   1.606  %As the approach relies on information available to the runtime, we expect that even better results will be observed for ``high-level'' parallel languages that more closely match application concepts instead of hardware concepts.
   1.607 @@ -1345,3 +1465,4 @@
   1.608  
   1.609   trying to pinpoint the causes of performance loss,  internal structure of 
   1.610  
   1.611 + involves  tuning both sequential behavior, inside units of work, and also 
   1.612 \ No newline at end of file

     2.1 --- a/0__Papers/Holistic_Model/Perf_Tune__long_version_for_TACO/latex/Holistic_Perf_Tuning.tex.orig	Sat Aug 03 19:24:22 2013 -0700
     2.2 +++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
     2.3 @@ -1,1249 +0,0 @@
     2.4 -%-----------------------------------------------------------------------------
     2.5 -%
     2.6 -%               Template for sigplanconf LaTeX Class
     2.7 -%
     2.8 -% Name:         sigplanconf-template.tex
     2.9 -%
    2.10 -% Purpose:      A template for sigplanconf.cls, which is a LaTeX 2e class
    2.11 -%               file for SIGPLAN conference proceedings.
    2.12 -%
    2.13 -% Guide:        Refer to "Author's Guide to the ACM SIGPLAN Class,"
    2.14 -%               sigplanconf-guide.pdf
    2.15 -%
    2.16 -% Author:       Paul C. Anagnostopoulos
    2.17 -%               Windfall Software
    2.18 -%               978 371-2316
    2.19 -%               paul@windfall.com
    2.20 -%
    2.21 -% Created:      15 February 2005
    2.22 -%
    2.23 -%-----------------------------------------------------------------------------
    2.24 -
    2.25 -
    2.26 -\documentclass[preprint]{sigplanconf}
    2.27 -
    2.28 -% The following \documentclass options may be useful:
    2.29 -%
    2.30 -% 10pt          To set in 10-point type instead of 9-point.
    2.31 -% 11pt          To set in 11-point type instead of 9-point.
    2.32 -% authoryear    To obtain author/year citation style instead of numeric.
    2.33 -\usepackage{amssymb,graphicx,calc,ifthen,subfig,dblfloatfix,fixltx2e}
    2.34 -
    2.35 -
    2.36 -% correct bad hyphenation here
    2.37 -\hyphenation{op-tical net-works semi-conduc-tor}
    2.38 -
    2.39 -\usepackage{wasysym}
    2.40 -
    2.41 -\begin{document}
    2.42 -
    2.43 -\bibliographystyle{plain}
    2.44 -%
    2.45 -
    2.46 -\conferenceinfo{WXYZ '05}{date, City.} 
    2.47 -\copyrightyear{2012} 
    2.48 -\copyrightdata{[to be supplied]} 
    2.49 -
    2.50 -\titlebanner{}        % These are ignored unless
    2.51 -\preprintfooter{Performance Tuning Using Semantic Information from the Language Runtime}   % 'preprint' option specified.
    2.52 -
    2.53 -
    2.54 -%MOIRAI: MOdel for Integrated Runtime Analysis through Instrumentation
    2.55 -\title{Integrated Performance Tuning Using Semantic Information Collected by Instrumenting the Language Runtime}
    2.56 -
    2.57 -\authorinfo{Nina Engelhardt}
    2.58 -           {TU Berlin}
    2.59 -           {nengel@mailbox.tu-berlin.de}
    2.60 -\authorinfo{Sean Halle}
    2.61 -           {Open Source Research Institute}
    2.62 -           {seanhalle@opensourceresearchinstitute.org}
    2.63 -\authorinfo{Ben Juurlink}
    2.64 -           {TU Berlin}
    2.65 -           {b.juurlink@tu-berlin.de}
    2.66 -
    2.67 -
    2.68 -% This \maketitle command is required from ieeepes version 4.0, to make
    2.69 -% ieeepes work correctly with newer LaTeX versions.
    2.70 -\maketitle
    2.71 -
    2.72 -
    2.73 -\begin{abstract}
    2.74 -Performance tuning is an important aspect of parallel programming. Yet when trying to pinpoint the causes of performance loss, often times insufficient knowledge  of the internal structure of the application and the runtime is available to understand how the observed patterns of performance have come to pass.
    2.75 -A trend in parallel programming languages is towards models that capture more structural information about the application, in an effort to increase both performance and ease of programming. We propose using this structural information in  performance tuning tools to make the causes of performance loss more readily apparent.
    2.76 -Our work produces a universal, adaptable set of performance visualizations that integrates this extra application structure, via a new model of parallel computation. The visualizations clearly identify idle cores, and tie the idleness to causal interactions within the runtime and hardware, and from there to the parallelism constructs that constrained the runtime and hardware behavior, thereby eliminating guesswork.
    2.77 -This approach can be used to instrument the runtime of any parallel programming model without modifying the application. As a case study, we applied it to the SSR message-passing model, and we walk through a tuning session on a large multi-core machine to illustrate how performance loss is identified and how hypotheses for the cause are generated. 
    2.78 -\end{abstract}
    2.79 -
    2.80 -
    2.81 -
    2.82 -
    2.83 -
    2.84 -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    2.85 -\section{Introduction}
    2.86 -\label{sec:intro}
    2.87 -
    2.88 -Performance visualization and tuning tools for parallel programs are critical to achieving good performance, and yet the experience of using current solutions leaves something to be desired. Often, performance tuning requires interpreting abstract views of usage or statistics, trying to \emph{guess} the cause of performance loss. Despite many different views of performance measurements, such as frequency by line of code, message sends and receives, and core usage timelines, the user does not know why a function runs in a particular spot on a particular core, nor whether that is desired behavior or erroneous behavior.
    2.89 -
    2.90 -Examining these myriad views can feel like being advised by the fabled six blind men examining an elephant by touching the part nearest to each\cite{PerfToolPoem}: one, reaching the side, compares it to a wall; the second, feeling the tusk, likens it to a spear; the next, touching the trunk, is reminded of a snake, and so on. All of these views are indeed correct and supply important information, but they don't connect to each other, to provide a view of how the parts interact to form a whole.
    2.91 -
    2.92 -Likewise, current tools may cover all the parts of the application code, but they often do not  adequately connect their observations to the runtime behavior, scheduling decisions, and consequent hardware behavior. The decision about which task or thread is assigned to which core at what point in time is at the heart of parallel performance, so these choices need to be displayed and connected to the application features influencing them.
    2.93 -
    2.94 -Our approach provides a more complete picture by following a new model of parallel computation. The model suggests that the relevant application features are the units of work performed and the  constraints on scheduling them, and that it is crucial to record and display the scheduling decisions taken by the runtime. The model simplifies hypothesis generation by connecting the measurements to each other and to the application, analogous to providing a view of the whole elephant.
    2.95 -
    2.96 -Our instrumentation, based on the model, detects the  units within the application code when they are scheduled during a run, and visually links each to the units upon which its execution is conditional. The visual arrangement makes identifying idle cores immediate, enables quickly generating the correct hypotheses for the causes of the performance losses, and quickly pinpointing where in the application code changes can be made to improve performance. In effect, the visualization serves as a map showing idle cores linked back to the section of application code responsible.
    2.97 -
    2.98 -The model of computation and our visual approach to using it is illustrated with a case study of performance tuning a standard parallel application on a large multi-core system.
    2.99 -
   2.100 -We start with an overview of previous approaches in Section \ref{sec:related}. We show usage of our visualizations through a case study in Section \ref{sec:casestudy}, and then expand on the model behind it in Section \ref{sec:theory}. Section \ref{sec:Implementation} will relate the model to implementation details. Finally, we conclude in Section \ref{sec:conclusion}.
   2.101 -
   2.102 -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   2.103 -\section{Background and Related Work}
   2.104 -\label{sec:related}
   2.105 -
   2.106 -When choosing a performance tuning tool, three primary factors to consider are:
   2.107 -\begin{itemize} 
   2.108 -\item the ease of recognizing performance loss
   2.109 -\item the ease of forming a hypothesis to explain the loss
   2.110 -\item the ease of linking the hypothesis to changes to make in application code
   2.111 -\end{itemize}
   2.112 -
   2.113 -The mental model that underlies the tool plays a key role in these three usefulness factors. This can be seen  by examining the  activity of performance tuning.  It is an iterative activity. To start, the programmer takes measurements during execution and compares them to the desired outcome. A hypothesis is generated for what caused any differences, by using a mental model, of causal relationships within  execution. The mental model is then also used to link the hypothesis to code, to suggest a change to make to improve performance. After modification, the code is run again, and these steps are repeated until the programmer is satisfied with the performance of the program. Thus, the mental model is central during performance tuning.
   2.114 -
   2.115 -As a result of its role when using  a tuning tool, the mental model underlying the tool is   a major focus when comparing  tools.  The mental model of our approach is based on a new model of parallel computation and is the main way  our approach differs. The new model makes it  easier to form a hypothesis of the cause of performance losses, and easier to link that back to changes to make in application code.
   2.116 -
   2.117 - Historically, the dominant mental model for parallel computation has been the software-thread abstraction. The model is very close to the hardware, without much higher-level semantics. This makes it  a problem when analyzing parallel performance. Yet as the earliest and most widespread parallel programming model, and one that serves as the basis on top of which many other models are implemented, it must be supported.
   2.118 -
   2.119 -A popular version is  pthreads, which introduces randomness through OS time-slicing, and whose synchronization constructs  imply only indirect connections. For instance, very little conclusion can be drawn about the relationship between the computations in two separate threads from their accessing the same lock consecutively. Consequently, tools such as Paradyn \cite{PerfToolParadyn} or VTune \cite{PerfToolVTune} that rely on this model  have a hard time connecting measurements to the application code details. They collect a wealth of statistics, but the application is to the tool a foreign process, where things ``just happen'' without causal links between events and code. Such tools generate conclusions such as ``application bottleneck is synchronisation'', and  ``too much time is spent in spinlocks'', without help as to what in the code causes this. For example, if there is more than one use of spinlocks in the application, it is not clear what point in the code is the problem.
   2.120 -
   2.121 -One fix to these issues is to allow the users to introduce measuring points into their own code, at the cost of increased programmer effort.  Because instrumentation code is written in the source language, it has access to application concepts. This advantage can be kept with automated instrumentation, by providing an instrumenting compiler, like the Tau \cite{PerfToolTau} project does.
   2.122 -
   2.123 -However, as long as the underlying execution model is still threads, there is no meaningful structure common to all applications to use to generate expressive measurement quantities. Function boundaries and the call graph are not sufficient when parallel execution is performed. The sequence and frequency of function calls tell little about parallel performance impacts because they have no bearing on synchronization events.  Unfortunately, pthreads does not capture even hints as to \emph{why} a given function call ends up blocking or not blocking, and what the effects on other threads are.
   2.124 -
   2.125 -Higher level parallel abstractions help alleviate that particular problem. For instance, in an application with MPI message passing \cite{MPI}, the information ``thread 2 spends little time waiting for messages from thread 0 but a lot of time waiting for messages from thread 1'' can be recorded, where in pthreads only ``thread 2 spends a lot of time waiting for a signal'' would be visible. It is much easier to reach the conclusion that the bottleneck is the slow rate at which thread 1 produces data.
   2.126 -
   2.127 -Leveraging the MPI library is a well-trodden path, with many tools available \cite{PerfToolParaver} \cite{PerfToolVampir} \cite{PerfToolParadyn} \cite{PerfToolTau}. But a frequent complaint of users of these tools is that in complex programs, the large number of messages sent makes it difficult to see problems. As there is little that distinguishes messages besides size, it is difficult to implement good filtering. Once again, we find that the limitations come from a parallel model that captures insufficient information.
   2.128 -
   2.129 -Fortunately, parallel languages are evolving in the same direction, and it is widely believed that semantically rich parallel constructs, when they match the structure of the application well, are key to improved productivity. With languages such as CnC \cite{CnCInHotPar} and StarSs \cite{StarSs}, the information available to the language runtime offers a much clearer picture of the constraints  placed on the execution of tasks and where they come from. Consequently, the application structure can be well reflected in visualizations, as the Temanejo debugger  \cite{PerfToolStarSs} for StarSs demonstrates.
   2.130 -
   2.131 -In the following section, we will demonstrate how our approach overcomes the various challenges of these other approaches through a combination of the following features:
   2.132 -\begin{itemize}
   2.133 -\item  it visualizes all the factors influencing scheduling decisions, given sufficient language and runtime pre-requisites
   2.134 -\item it has a filtering mechanism that shows only the most relevant groups of constraints, as desired
   2.135 -\item it has flexibility to capture and represent any type of constraints
   2.136 -\item it works for any parallel programming library or language
   2.137 -\item it instruments only the runtime, once,  for all applications
   2.138 -\end{itemize}
   2.139 -
   2.140 -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   2.141 -\section{Case Study: Analyzing an SSR Application}
   2.142 -\label{sec:casestudy}
   2.143 -
   2.144 -To concretely illustrate the benefits of connecting measurements to an effective computation model, in this section we walk through a typical performance tuning session using our approach. This shows the features  in action, and indicates how having those features provide an advantage to the user over methods that lack them.
   2.145 -
   2.146 -We first describe the program and language used, and then the features of our visualization. After this preparation, we show a sequence of the visualizations. In each,  we point out how the  performance loss is identified, and which visual features suggest the hypothesis of the cause of the loss.  We show how the visual features direct the user to the specific sections of code that need to be changed, and how the model suggests what changes to try. 
   2.147 -
   2.148 -\subsection{Application and Target Hardware}
   2.149 -
   2.150 -In our session, we wish to tune a standard program that the reader  knows well. The best example is likely matrix multiply, with which the reader should be familiar, allowing concentration on the tool without distraction about the application. 
   2.151 -
   2.152 -We run it on a machine with 4 sockets by 10 cores each, for a total of 40 physical cores. They are Intel WestmereEx cores running at 3.0GHz, with TurboBoost turned off for reproducability. 
   2.153 -
   2.154 -The application code uses language features to create virtual processors (VP). The first VP created divides the work into a number of pieces and creates a new VP for each piece. 
   2.155 -
   2.156 -How many pieces is set by a tuning parameter in the code, and the number of cores. The application uses language features to distribute the VPs across the cores,  in a round-robin fashion unless otherwise specified.
   2.157 -
   2.158 -It then creates a results VP that receives a partial-result from each piece and accumulates the results. The  divider VP  then waits for the results VP to indicate completion, after which the language runtime shuts down.
   2.159 -
   2.160 -\subsection{Programming Model}
   2.161 -The language used is SSR, which is based on rendez-vous style send and receive operations made between virtual processors (VPs), which are equivalent to software threads. The example application uses commands for creating and destroying VPs, and  two kinds  of  send-receive paired operations. 
   2.162 -
   2.163 -The first, \emph{send\_from\_to} specifies both sender and receiver VPs. The results VP uses this to tell the divider VP that the work is complete. The second, \emph{send\_of\_type\_to}, specifies only a specific receiver, leaving the sender  anonymous. It has increased flexibility while maintaining some control over scope. The worker VPs use this  to send their partial-result to the results VP. 
   2.164 -
   2.165 -The language also includes a \emph{singleton} construct that designates a piece of code as to be executed only once. It was used to shift work out of the divider, which is sequential, over to the worker-pieces, which spreads the work across many cores.  The work is setup that many pieces share but is only to be performed once. 
   2.166 -
   2.167 -Miscellaneous performance constructs are also available, such as one to force which core a virtual processor is assigned to. We use this in our example program to control scheduling.
   2.168 -
   2.169 -A note on terminology: We often use the term  ``work-unit'',  which we define precisely, instead of ``task'', which has acquired multiple  meanings in the literature. Work-unit  is defined as the trace-segment performed on a core, between two successive scheduling events, plus the set of datums consumed by that trace segment. The word task often maps well onto this  definition, and we use both words, but mean the precise work-unit definition when we say task.
   2.170 -
   2.171 -
   2.172 -
   2.173 -\subsection{The Visualizations}
   2.174 -\label{subsec:visualization_def}
   2.175 - The first visualization is what we refer to as a  Unit \& Constraint Collection, or UCC. It depicts the constraints on the scheduling decisions that come from the application. Constraints can be dependencies stated explicitly in the code, or can be implied by language constructs. These limit the choices the runtime is allowed to make. 
   2.176 -
   2.177 -The second visualization we call  the scheduling consequence graph (SCG), or just consequence graph (CG). It depicts the scheduling operations performed by the runtime, and the consequent usage of the cores. 
   2.178 -
   2.179 -The UCC shows only application-derived information, as opposed to the consequence graph, which combines the \textit{use} of the UCC-depicted constraints along with runtime-imposed dependencies and hardware-imposed constraints.  Hence, the UCC states the degrees of freedom enabled by the application, while the consequence graph states how those were made use of, by a particular runtime on particular hardware.
   2.180 -
   2.181 -\subsubsection{UCC visualization}
   2.182 -\begin{figure}[ht]
   2.183 -  \centering
   2.184 -  \includegraphics[width = 3.2in, height = 1.8in]{../figures/UCC_stylized_for_expl.pdf}
   2.185 -  \caption{Two versions of a stylized Unit and Constraint Collection (UCC). Both display the  logical connections among units, but the right UCC adds an indication of the amount of work in a unit.}
   2.186 -  \label{fig:UCC_expl}
   2.187 -\end{figure}
   2.188 -
   2.189 -Fig. \ref{fig:UCC_expl} shows two versions of a UCC, both stylized for purposes of explanation.  These are used less often than the SCG,  mainly as a helper when brainstorming ways to get more usable parallelism out of an application. 
   2.190 -
   2.191 -Both versions of the UCC depict constraints on units. Starting  at the top, they  progress downwards,  placing units that constrain  above those constrained by them, and connecting them by a line representing the constraint. A unit is placed into the row just below the last of  the units constraining it, in the first available column. However, the rows and columns are just visual conveniences; the only information in the UCC is the lines between units, which represent constraints among the units. In practice, each unit is also labeled with a unique unit ID, from which the code of the unit and the parameters passed to it can be looked up. 
   2.192 -
   2.193 -What distinguishes the two types of UCC is the size of the units. The left one shows only the logical graph-like structure of the application, while the right weights the height of a unit by the number of instructions executed in it. When information about unit size is available, it can visually indicate the \textit{useable} parallelism available.
   2.194 -
   2.195 -\subsubsection{SCG visualization}
   2.196 -\begin{figure}[ht]
   2.197 -  \centering
   2.198 -  \includegraphics[width = 2in, height = 1.8in]{../figures/SCG_stylized_for_expl.pdf}
   2.199 -  \caption{Stylized Scheduling Consequence Graph (SCG).}
   2.200 -  \label{fig:SCG_expl}
   2.201 -\end{figure}
   2.202 -
   2.203 -Fig. \ref{fig:SCG_expl} shows a consequence graph, stylized for purposes of explanation. This is the main visualization used during performance tuning, because it shows the consequent uses of the cores in the machine, and links those to the constraints and code in the application. 
   2.204 -
   2.205 -It is composed of a number of columns, one for each core. A column represents time on the core,  increasing as one goes down, measured in clock cycles. It is broken into blocks, each representing the time accounted to one work-unit. Each block is further divided into regions, each a different color, which indicates the kind of activity the core was engaged in during that  region's time-span.
   2.206 -
   2.207 -The application code executed within a block is linked to the block. In our tool, the block is labelled with a unique unitID. This ID  is then linked to the code executed within that unit. In this way, the code of any block can be looked up, along with the parallelism constructs that mark the start and end of the block.
   2.208 -
   2.209 -The kinds of activities within a block are defined by the computation model that underlies the visualization. The first kind of activity is the actual work, plus  waiting for  cache misses. It is represented by a blue-to-red region where the color indicates  intensity of cache misses, with pure red representing at or above the maximum misses per instruction, and pure blue the minimum (the max and min are set in the tool that generates the visualization).
   2.210 -
   2.211 -  The second kind of activity is runtime overhead, represented by a gray region. This is the overhead spent on that particular work-unit. When desired by the user, it is further broken into pieces representing activities inside the runtime. The options include time spent on: constraints, when determining readiness of the work-unit; deciding which ready unit to assign to which hardware; and time spent switching from virtual processor, to the runtime, and back. In this paper, we show all runtime overhead lumped together, however in other circumstances a breakdown can be key to understanding interaction between runtime and application. 
   2.212 -
   2.213 -The other type of visual feature seen in Fig. \ref{fig:SCG_expl} is lines. Each represents a construct that influenced scheduling, where the color indicates what kind of construct.   A line represents two things: a constraint, whose satisfaction made the lower unit ready, and a decision by the runtime to start the lower unit on that core. 
   2.214 -
   2.215 -In general, lines may also be drawn that represent other kinds of interactions, which affect core usage. For example,  our runtime implementation only allows one core at a time to access shared  scheduling state. Visualization of this can be turned on, as additional lines linking the gray runtime regions of blocks (visualization of such interactions is turned off in this paper). 
   2.216 -
   2.217 -Two work-unit blocks   that appear in sequence and have no lines drawn to them often have a causal dependency between them, due to the semantics of the base language. Visualization of these dependencies is also turned off, but can be inferred via the link to the code.
   2.218 -
   2.219 -Note that  many different orderings can be validly chosen. Which scheduler choices are valid is determined by three kinds of constraints: the application code constraints, hardware constraints, and runtime implementation imposed constraints.
   2.220 -
   2.221 -The visual features allow the user to see at a glance the total  execution time (height), idle cores during the run (empty columns), cache behavior (color of work regions), degree of overhead (size of gray regions), and which units constrained which other units (lines). All consequence graphs in this paper are at the same scale, so they can be compared directly.
   2.222 -\begin{figure*}[t!]
   2.223 -  \begin{minipage}[b]{0.25\textwidth}
   2.224 -        \hfill\subfloat[35.8 Gcycles\\Original]
   2.225 -    {\quad\quad\includegraphics[scale=0.015]{../figures/222.pdf} \quad
   2.226 -    \label{fig:story:a}}\quad\hspace{10px}
   2.227 -  \end{minipage}
   2.228 -  \begin{minipage}[b]{0.7\textwidth}
   2.229 -  % \subfloat[]
   2.230 -  %  {\includegraphics[scale=0.015]{../figures/194.pdf} 
   2.231 -  %  }\quad
   2.232 -   \subfloat[13.0 Gcycles\\ after fixing the load balancer]
   2.233 -    {\includegraphics[scale=0.015]{../figures/2.pdf} 
   2.234 -    \label{story:b}}\quad
   2.235 -   \subfloat[11.0 Gcycles\\ after changing so as to put work on core 1 first (solution 1)]
   2.236 -    {\includegraphics[scale=0.015]{../figures/5.pdf} 
   2.237 -    \label{story:c}}\quad
   2.238 -  % \subfloat[S1+divide factor 0.2]
   2.239 -  %  {\includegraphics[scale=0.015]{../figures/208.pdf} 
   2.240 -  %  }\quad
   2.241 -  % \subfloat[S1+divide factor 0.3]
   2.242 -  %  {\includegraphics[scale=0.015]{../figures/209.pdf} 
   2.243 -  %  }\quad
   2.244 -   \subfloat[10.9 Gcycles\\ after also changing the divide factor from 0.6 to  0.5]
   2.245 -    {\includegraphics[scale=0.015]{../figures/6.pdf} 
   2.246 -    \label{story:d}}\quad
   2.247 -   \subfloat[15.6 Gcycles\\ after further changing the divide factor to 0.4]
   2.248 -    {\includegraphics[scale=0.015]{../figures/7.pdf} 
   2.249 -    \label{story:e}}\quad\\
   2.250 -   \subfloat[10.4 Gcycles\\ after going back to put divider VP onto its own core (Solution~2)]
   2.251 -    {\includegraphics[scale=0.015]{../figures/12.pdf} 
   2.252 -    \label{story:f}}\quad
   2.253 -   \subfloat[10.3 Gcycles\\ after then moving the receive VP to the same core as the divider VP]
   2.254 -    {\includegraphics[scale=0.015]{../figures/10.pdf} 
   2.255 -    \label{story:g}}\quad
   2.256 -   \subfloat[9.7 Gcycles\\ after additionally changing the divide factor to 0.4]
   2.257 -    {\includegraphics[scale=0.015]{../figures/15.pdf} 
   2.258 -    \label{story:h}}\quad
   2.259 -  % \subfloat[S2+divide factor 0.3]
   2.260 -  %  {\includegraphics[scale=0.015]{../figures/217.pdf} 
   2.261 -  %  }\hspace{2in}
   2.262 -  % \subfloat[scheduling issue with singleton]
   2.263 -  %  {\includegraphics[scale=0.015]{../figures/194.pdf} 
   2.264 -  %  }\quad
   2.265 -  \end{minipage}
   2.266 -\caption{Performance tuning with Scheduling Consequence Graphs (all figures drawn to the same scale)  }
   2.267 -\label{story}
   2.268 -\end{figure*}
   2.269 -
   2.270 -\subsection{Walk-through}
   2.271 -\label{subsec:walk-through}
   2.272 -
   2.273 -The scheduling consequence graphs that display the measurements collected on various runs during tuning are displayed in Fig. \ref{story}. All use the same scale, for direct comparison. They have 40 columns, one for each core, and relative height indicates relative execution time. The lines in red, orange, and green represent application-code constructs. Red is creation of a virtual processor, green is the many-to-one \texttt{send\_of\_type\_to}, and orange is the singleton construct. For better visibility, only constraints that cross cores are shown.
   2.274 -
   2.275 -After functional debugging, the first tuning run produces the consequence graph seen in Fig. \ref{fig:story:a}. The first thing to notice is that it is slimmer than expected: of the 40 available cores, only 13 are being used. Because the application places work on cores explicitly, this must be a bug in the dividing code. A cursory inspection reveals that a closing curly brace in the distribution loop had been misplaced. This may be a very simple bug, but it went unnoticed despite using this application as a test program for development of the language runtime (while using traditional thread-based performance tools).
   2.276 -
   2.277 -\subsubsection{Second Run}
   2.278 - After fixing this, the next run (Fig. \ref{story:b}) corresponds much more to the expected execution behaviour. However, there remains a noticeable section at the beginning where only 3 cores have work and the other 37 remain idle.
   2.279 -
   2.280 -Zooming in on those cores, we see that creation code starts running on core 0, within the creation VP, and then the next block on the core is work. Creation stops, starving the other cores. Looking at the creation code, we see that the creation VP assigns the first work VP to its own core, so that work is now waiting in the queue to execute there. When it creates the second work VP, that creation call switches core 0 to the runtime. When done with creation, the runtime takes the next VP from the queue, which is that waiting work VP. Hence core 0 does the work next instead of continuing with creation (the merits of work stealing or other scheduling strategies are independent from this illustration of how to use this approach to performance tune).
   2.281 -
   2.282 -The hypothesis was generated by looking at the code linked to each block and noting the visual pattern that creation code stopped running on core 0. Work code started running instead, and only after it finished did creation code start again. Hence, visual cues led directly to the hypothesis. 
   2.283 -
   2.284 -Two solutions come to mind: assign work to the other cores first, so that they would be busy when the creator VP's core switches to work, or else dedicate a core to the creator VP. The first solution has the advantage of preserving performance of the application even when run on a machine with a single-digit number of cores, so we tried it first. 
   2.285 -
   2.286 -\subsubsection{Third run} Assigning work to the other cores first gives us Fig. \ref{story:c}. The section that was at the top, with idle cores, has disappeared. However, an idle period can still be observed between the first and the second set of work units. The work units have roughly the same length and  the work on core 0 starts last, so finishing the core 0 work holds up creation until after the other cores have finished their work (note that work on some cores takes slightly longer because that core performs the copy-transpose singleton, and also variations are caused by cache misses).
   2.287 -
   2.288 -  It is also noticeable that in the second set of work units, not enough work pieces remain to fill all cores. 16 out of 40 remain idle at the bottom.
   2.289 -
   2.290 -\subsubsection{Fourth and fifth runs} To try to fill the empty columns at the end, we modified the size of the work units. However, as figures \ref{story:d} and \ref{story:e} show, this did not help. The blank areas between ``bands'' of work can be seen by the red lines to be due to  creation. The increased number of units causes creation to be the bottleneck again, and the time lost between sets grows larger than the time that previously was lost.
   2.291 -
   2.292 -\subsubsection{Sixth run} At this point we wanted to try the road not chosen, dedicating a core to the creation VP.
   2.293 -Going back to version b of the code and implementing this solution, instead, leads to Fig. \ref{story:f}. The blank area between the two sets has disappeared, showing a 4\% shorter execution time.
   2.294 -
   2.295 -\subsubsection{Seventh and eighth runs}Core 0 is now empty after the creation phase at the beginning, so we  moved the receive VP there (Fig. \ref{story:g}). This only minimally improved the time, at this size of work unit, but allows overlapping the result collection with other work, which is an advantage when cutting  the work into more pieces, requiring longer collection as in Fig. \ref{story:h}.
   2.296 -
   2.297 -Overall it is also noticeable that as work units become smaller, execution time becomes more irregular. Variability in work length correlates with the color, indicating cache behavior has worsened with smaller work size.
   2.298 -
   2.299 -%Note that the hypothesis that cache behavior worsened with smaller work sizes, was generated directly from visual cues.    
   2.300 -
   2.301 -\subsubsection{Holes in the core usage}
   2.302 -
   2.303 -In Fig \ref{story:d}, ``holes'' are noticeable. Inspecting these holes closer, we can see that the stalled blocks are at the end of orange lines. This indicates they are waiting upon the completion of a singleton. The pattern of blocks shows that usually  the singleton unit runs before the work unit, but in these cases the singleton code was delayed until after the work on that core.  This is a runtime implementation fluke. The only thing an application programmer can do is change the work size to minimize the impact. (For those curious, the first VP to reach the singleton is granted control, but a ready work VP lands in the queue during the granting activity, so when the runtime finishes granting, the work VP is next, and the VP that now owns the singleton sits and waits for the work to end. All work VPs on other cores that pass through the same singleton  also wait.)  
   2.304 -
   2.305 -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   2.306 -\section{The Model Behind the Visualization}
   2.307 -\label{sec:theory}
   2.308 -Now that the usage has been seen, we expand on the model behind the visualizations. The model ties the information together, and understanding it helps in generating  hypotheses from the visualization features. 
   2.309 -
   2.310 -As seen, the model has two parts, a \emph{Unit \&\ Constraint Collection (UCC)}, and a \emph{Scheduling Consequence Graph} (SCG or just consequence graph,  CG).  The UCC indicates the scheduling  choices  the application allows, and so shows what the programmer has control over. The consequence graph says which of those were actually taken during the run and the consequences of that set of choices.
   2.311 -
   2.312 -We give a more precise description of UCC, then consequence graph, in turn.
   2.313 -However, space is too limited for a complete definition, which is given in a companion paper submitted to a longer format venue.
   2.314 -\subsection{Unit \& Constraint Collection}
   2.315 -The UCC contains all the units of work that get scheduled during a run, and all constraints the application places on scheduling those units. This is the simple definition, but unfortunately, this information is not always easy to obtain. The complication is that different classes of application exist, with two degrees of freedom that determine how much of the UCC is actually defined in the application vs the input data, or even in the runtime.
   2.316 -
   2.317 -Some applications have everything determined in the code, with all units fixed, and all constraints fixed. An example is matrix multiply with fixed size matrices.  But for others, the shape of the UCC is only partially defined by the application code.  Take matrix multiply used in Section \ref{sec:casestudy}, where an input parameter determines the number of units created. Here, the UCC is different for each parameter value. An extreme example is an NP complete problem, with redividable units, for which the units are a function of both the  input data \emph{and} decisions made by the runtime.
   2.318 -
   2.319 -\begin{figure}[ht]
   2.320 -  \centering
   2.321 -  \includegraphics[width = 3.2in, height = 1.8in]{../figures/UCC_weighted.pdf}
   2.322 -  \caption{Concrete UCC measured for matrix mulitply during the performance tuning in Section \ref{sec:casestudy}.}
   2.323 -  \label{fig:UCC_example}
   2.324 -\end{figure}
   2.325 -
   2.326 -    We call a fully specified UCC a \emph{concrete} UCC.  Every run of an application eventually winds up defining a concrete UCC, such as produced for the performance tuning, as seen  in Fig. \ref{fig:UCC_example}. In it, every unit that was scheduled in the SCG appears,  along with  the application-defined constraints on scheduling them.  For this application, parameters determined the work  division, and so determined the units. Hence, the application alone does not specify the concrete UCC, because the units remain unknown until parameter values are given. In general, the amount of UCC made concrete by the application alone falls into a two-dimensional grid. One dimension covers the units, the other the constraints.
   2.327 -
   2.328 -\begin{figure}[ht]
   2.329 -  \centering
   2.330 -  \includegraphics[width = 2in, height = 1.8in]{../figures/UCC_concreteness_grid.pdf}
   2.331 -  \caption{Abstract representation of the amount  of concreteness of a UCC. The letters A, B, C, D stand for UCCs described in the text.}
   2.332 -  \label{fig:UCC_Concreteness}
   2.333 -\end{figure}
   2.334 -
   2.335 -Fig. \ref{fig:UCC_Concreteness} shows the two axes and the four sets of additional information that have to be added to the UCC in order to determine the units and constraints in the final fully concrete UCC. The position an application-derived UCC, or later in the life-line UCC, lands on the grid indicates how far it is from being fully concrete.  The horizontal   position indicates what inputs are still needed to determine the units, and vertical the constraints.  0 indicates that the units (constraints) are fully determined by the application code alone; 1 means parameter values also must be known; 2 means input data values also play a role, and 3 means the units (constraints) can only become known  after runtime scheduling decisions have been made. 
   2.336 -
   2.337 -The closer an application-derived UCC is to the origin, the less additional information is needed to obtain a  concrete UCC descendant of it. The UCC labeled A in the figure is fully concrete just from the source code alone (representing for example, matrix multiply with fixed size matrices and fixed division). The UCC labeled B requires the input data and parameters to be specified before its units are concrete, but just parameters to make its constraints fully concrete (as per ray-tracing, with bounce depth specified as a parameter). The UCC labeled C only has variability in its constraints, which require input data (for example, H.264 motion vectors).
   2.338 -But even the least concrete UCC, out at the end of the diagonal (D in the figure), generates a concrete descendant during a run of the application.
   2.339 - 
   2.340 -Notice, though, that even a fully concrete UCC still has degrees of freedom, in deciding which units to run on which hardware and in what order of execution. These decisions fix interactions within the hardware, to yield the communication patterns and consequent performance seen during the run. 
   2.341 -
   2.342 -An added twist is that an application has a life-line, spanning from code all the way through the run, and its representation  may change at the different stages of life. It starts as pristine source, then moves  into specialization where code is translated into different representations than the original, and finally the specialized code is run. A different version of UCC may be generated at each of these points in  the life-line.
   2.343 -
   2.344 -For example, specialization may perform a static scheduling, which fixes the units, moving the UCC towards the origin. Alternatively, the toolchain may inject manipulator code for the runtime to use, which lets it divide units during the run when it needs more. The injection of manipulator code makes the UCC less concrete, moving it further from the origin.
   2.345 -
   2.346 -The progression of UCCs generated during the life-line indicate what entity has the control over which units and constraints appear in the final concrete UCC. Viewing the progression gives insight into what is inside the application programmer's control vs under control of each tool in the toolchain or  the runtime. For example, the original application-derived UCC  indicates what can be done statically: the further out on the diagonal that UCC is, the less scheduling can be done statically in the toolchain.
   2.347 -
   2.348 -In this paper, we do not suggest how to represent UCCs far out on the diagonal. One of those actually indicates a multi-verse of concrete-UCCs. Which of them materializes  depends on the data and what the scheduler decides. We only represent the concrete UCC that materializes during a run and leave the question of representing less concrete ones to future work.
   2.349 -
   2.350 -\subsection{Scheduling Consequence Graph}
   2.351 -Whereas the UCC concentrates on application-derived information, the second part of the model adds-in effects of runtime  details and hardware. We called it the Scheduling Consequence Graph because it links scheduling decisions to their performance consequences. But it also indicates the role, in the decision, of application, runtime and hardware details.
   2.352 -As a result it identifies instances of lost performance, and links them to the cause of the loss, as seen in Section \ref{sec:casestudy}.
   2.353 -
   2.354 -Contrary to the UCC, the consequence graph shows the behavior resulting from scheduling decisions actually \emph{made},   from among those \emph{possible.} The UCC shows just the possibilities. Hence, a consequence graph shows \emph{one} of the possible choice-sets allowed by the UCC.
   2.355 -
   2.356 -A consequence graph accounts for each bit of core time. It has boxes and arcs, with the boxes divided into regions. The boxes  each represent all core time assigned to one work unit, with each region inside representing a segment of time that the core was engaged in a specific type of activity.  An arc links  regions (or boxes)  and represents a causality of some kind.
   2.357 -
   2.358 -There is one kind of region for each reason that the core is being used (or being forced idle), and several kinds of arcs, one for each type of causality between regions.
   2.359 -
   2.360 -The core activities of the region types are: application work, idling while waiting for communication of work data, managing constraints, choosing assignment of work onto cores, and runtime internals. The runtime internals in turn have sub-categories, which we will not detail here, but some of which will be mentioned in Section \ref{sec:Implementation}.
   2.361 -
   2.362 -There is an arc-type for each category of causality. The arc-types (causality categories) represent: sequential dependency in the base language being satisfied, parallel constraint being satisfied (i.e., one unit did something to satisfy a constraint on the other, causing it to be free to be scheduled), runtime internal causality such as a global lock (runtime on one core releases the lock, causing the other to acquire it), and arcs that represent hardware causal relationships (one work-unit finishing on a core causes another work-unit to start there, given the choice by the runtime). The formal details are given in the longer format companion paper.
   2.363 -
   2.364 -We will now look at each source of causal relationship.
   2.365 -
   2.366 -\paragraph{Constraint  causality} There is a constraint causality when  action by one unit causes (or contributes to) satisfaction of the constraint blocking the other unit. This includes sequential dependencies from the base language.
   2.367 -
   2.368 -Sequential dependencies may add superfluous constraints that  eliminate some otherwise allowed choices in the UCC. An example would be a \texttt{for} loop that creates work-units -- no parallelism constructs cause the creations to be done in sequence, but the base C language sequentializes it nonetheless. 
   2.369 -
   2.370 -\paragraph{Runtime internal causality} Runtime implementation details may introduce ``extra" causalities between units. For example, the version of VMS we instrumented for this paper runs separately on each core and relies upon a global lock for accessing shared runtime information. This lock introduces a causal relationship  when the runtime on one core is attempting to process one unit, but must wait for the runtime on a different core to finish with its unit.
   2.371 -
   2.372 - Normally, these are not displayed explicitly, due to clutter, but can be turned on when needed, for instance to determine the cause of a particular pattern of core usage.
   2.373 -
   2.374 -\paragraph{Hardware causality} The physical fact that a given resource can only be used by one work-unit at a time introduces hardware causalities. When multiple units are free to execute, but all cores are busy, then completion of a unit  on one core causes (in part) the next ready unit to run on that core. 
   2.375 -
   2.376 -These are also not normally displayed, due to clutter, and not all hardware dependencies are directly measured. Future work will focus on using the performance counters and other instrumentation to add more information about communication paths taken as a consequence of the scheduling decisions made. It will start with the current linkage of application-code to runtime decisions, and add consequent usage of communication hardware. This gives an end-to-end linkage between runtime choices and caused behavior on the hardware. 
   2.377 -
   2.378 -Consequence graph features each tie back to features in the UCC and thence to specific segments of code or constructs.
   2.379 -
   2.380 -\subsection{Levels of UCC and Consequence Graph}
   2.381 -There is one last twist to the story of UCCs and consequence graphs, which is that there are levels of them that correspond to the levels of scheduler in a hierarchical machine. We use an example involving a server machine with a hierarchy of runtimes to illustrate both, concentrating first on just the UCCs, then adding the consequence graph. 
   2.382 -
   2.383 -\subsubsection{Levels of UCC}
   2.384 -For the example, consider  a server with one rack,  having a back-plane that boards plug into. A board has its own memory with four sockets, each having a chip with four cores. So there is a back-plane network connecting the boards,  a bus on each board that connects the sockets to the DRAM, and inside the chip in each socket is a cache hierarchy that connects the cores.
   2.385 -
   2.386 -The hardware is given a set of runtimes to match the hierarchy. Each network or bus has a runtime that schedules work onto the things connected below it. So the top runtime divides work among the boards, while a board's  runtime  divides work among the sockets, and a socket's  runtime  divides work among the cores.  
   2.387 -
   2.388 -To a runtime high up, each runtime below it looks like a complete machine. It schedules work-units to those machines, without knowing the internal details of how that machine is implemented. So the runtime at the top handles very large work-units that it schedules onto the runtimes on the boards. A board-level runtime  divides up the work-unit it gets into smaller work-units, then schedules one onto each socket's runtime, and so on.
   2.389 -
   2.390 -The application in this example has been written in a language that allows work to be divided. The toolchain inserted a manipulator that allows each runtime to divide up the work it is given into smaller work-units, such as via the DKU pattern \cite{DKUTechRep_09}. This pushed the UCC of the application all the way to the right on the unit axis.  
   2.391 -
   2.392 -So what does the concrete UCC produced during a run look like? Well, a unit is defined as the work resulting from one scheduling decision. Each runtime has its own scheduler, which means units are defined for each runtime.  That in turn means that each runtime has its own concrete UCC.
   2.393 -\begin{figure}[ht]
   2.394 -  \centering
   2.395 -  \includegraphics[width = 3.2in, height = 1.8in]{../figures/UCC_Levels_2.pdf}
   2.396 -  \caption{Representation of multiple levels of  UCC.}
   2.397 -  \label{fig:UCC_Levels}
   2.398 -\end{figure}
   2.399 -
   2.400 -Fig. \ref{fig:UCC_Levels} shows that these UCCs are related to each other in the same hierarchy as the runtimes. A unit scheduled by one runtime is in turn broken into sub-units that exist at a lower level. Each of those units is  then separately scheduled by the lower-level runtime, according to a UCC just for them. So, as the figure shows,  a unit in one UCC has an entire lower-level UCC inside it.  
   2.401 -
   2.402 - %Great, that makes sense, now what about the consequence graphs?
   2.403 -
   2.404 -\subsubsection{Levels of Consequence Graph}
   2.405 -With multiple levels of scheduling activities, a scheduling consequence graph can be constructed for each level. 
   2.406 -
   2.407 -For a given level, a lower-level runtime is seen as a single processor entity. So, what is the consequence of deciding to schedule a given work unit onto one of those  processor entities? When the goal is  improving performance, the consequence of interest is the time taken until completion of the work unit.
   2.408 -
   2.409 -Hence, the question becomes,  when can the upper-level work unit be considered completed? The answer is, when all the lower-level units of which it is comprised are completed, plus any additional work spent dividing up the work and combining results. Once the lower-level consequence graph is established, this  time can be easily determined: it is the critical path across the lower-level consequence graph plus the additional, non-overlapped, time spent creating the lower-level units and combining results into the format of the higher-level unit. The divide and recombine are the creation of    the lower level, and so not part of it.     
   2.410 -
   2.411 -Consider the concrete example of the SSR matrix multiply application from Section \ref{sec:casestudy}. To start we go up a level, above the process created to execute the matrix multiply. Here, a unit is the execution of an entire application, invoked via OS commands. The  work-time of that unit is the execution time of the application, from the first instruction of the application code until the last. This time equals  the critical path through the matrix multiply work, plus creation of the various VPs, and collection of the results by the results VP. 
   2.412 -
   2.413 -If the case study had been performed in hierarchical fashion, then the SCG seen there would be divided between levels, with creation and results accumulation  moved to the higher application-level SCG. The lower-level would only contain the work of the multiply. 
   2.414 -
   2.415 -One twist is that the divide and recombine are themselves composed of  smaller work-units.  However, they are not visible to units in the work portion, although there are causalities between them. The divide units end with creation of work-level units, while the end of work-level units causes creation of recombine units.
   2.416 -Hence, divide and recombine are  inside of  the higher-level unit while not part of the lower-level SCG. They can be conveniently displayed in a single visualization because of the clean connection between levels provided by the create and end causalities.   
   2.417 -
   2.418 -To complete the discussion, we consider going down another level, into the physical processor.  In modern super-scalar multi-cores, each core is a parallel processor composed of functional blocks such as arithmetic or floating-point blocks, along with logic that issues instructions to those blocks. We first break down a single  SSR work unit of the kind seen in Section \ref{sec:casestudy}, into smaller work units, namely individual instructions. Then we break instructions down into pipeline stages.
   2.419 -
   2.420 -The shape of the UCC for the instructions  inside a single SSR level work-unit is a straight line of sequential dependencies. However, the pipeline stages inside a core modify the UCC. They analyse the register use dependencies between instructions and generate a new UCC consisting of  functions constrained by the data dependencies. The dependencies are enforced by the issue logic, which  goes on to assign ready functions to the functional blocks.  The choices made by the issue logic  determine the shape of the SCG for this level, and the critical path of that SCG is the work time of the higher, SSR level, work-unit.
   2.421 -
   2.422 -The pipestages spent performing fetch, decode, analysis,  issue, and retirement correspond to functions in a runtime.  Fetch corresponds to creation of the instruction-level work-unit. Decode and analysis logic generate the new UCC, as part of the runtime,  Issue logic  performs  the scheduling function of a runtime. Retirement is also part of enforcing the dependencies, and so is  also part of the runtime's scheduler.  Most of the overhead of these pipestages is overlapped by the pipeline effect. but  does still contribute to the critical path during pipeline disruptions like mis-predicted branches. It is counted as non-overlapped runtime overhead.   
   2.423 -
   2.424 -%To give insight into going from the SSR work-unit level up to the application-level, let us now go from the instruction level up to the SSR unit level, `looking up' so to speak. Considering overhead, from inside the core,  the SSR level overhead looks just like more instructions. 
   2.425 -
   2.426 -%Nina: we should talk this part over.. (I should have left more of your wording in place in comments..  I like this way of co-editing.. : )
   2.427 -
   2.428 -%One level up, the same holds true: The overhead of the unique application-unit would be the time spent in the OS, doing things such as setting up the application's memory space. This time is not visible from inside the application. However, it uses the same processing units (in this case, cores), and to them appears not very different from application work. The noticeable difference, in this case, is that the OS runtime is written using a different programming model instead of SSR.
   2.429 -
   2.430 -
   2.431 -%Consider, in the matrix multiply code, the core usage spent  while  dividing the work and handing it to other cores. This is not work of the application, but overhead spent breaking the single application-unit into multiple sub-units.   Even though it is in the application code, it's purpose is implementing the execution model, which makes it runtime overhead. But which runtime level? It's not part of the SSR language runtime, so not overhead of a unit inside the application,   but rather it's for the application itself, as a unit! So, the core time spent calculating the division  gets counted towards the application-level unit, while the time spent inside the SSR runtime creating the meta-units is counted towards those lower SSR-level units. But both are in the critical path, so both charged as work time of the higher-level unit.
   2.432 -
   2.433 -
   2.434 -% Another way to view this is that  by the process of elimination, the only core-time not accounted for elsewhere is the time spent dividing up a unit into smaller ones, and time spent accumulating the individual results back together. So this is what gets charged to the  higher-level unit.
   2.435 -
   2.436 -%The last question is how to handle communication consequences, which result from decisions made in all levels? The decisions in higher-level runtimes set the context for decisions in lower-level ones, which links a higher-level choice  to the consequences resulting from the lower-level choices. But the value of a consequence graph comes from linking the size of boxes in it to the decisions made by the scheduler. It's not clear how to divide-up  the time  cores spend waiting for non-overlapped communication, to assign portions to different levels. We have no good answer at the moment and leave it for future work.
   2.437 -
   2.438 -In practice, the question of levels does not come up very often for multi-core machines, as there is generally only one level of interest being investigated. However, the perspective is occasionally important here, e.g. when identifying the different components of the model in a concrete runtime so as to insert instrumentation code in the right places.
   2.439 -
   2.440 -
   2.441 -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   2.442 -\section{Implementation}
   2.443 -\label{sec:Implementation}
   2.444 -
   2.445 -Now that the usage and theory are in hand, we give the details of implementation. We attempt a bottom-up view, to provide an alternate path to understanding the model and visualizations,  as well to provide a path to derive the benefits directly from the details. 
   2.446 -
   2.447 -This section will show the level of effort needed to implement our approach for a new language. In short, this involves inserting collection points into the runtime, then modifying the post-processing that produces the visualization.
   2.448 -
   2.449 -We cast the implementation in terms of the computation model, then identify the
   2.450 -points inside the runtime that correspond to points in the model. It is only in these spots that instrumentation gets inserted into runtime code. 
   2.451 -
   2.452 -One benefit seen directly from these details is that many things remain the same across languages, because the computation model is invariant across languages.  This includes the basic nature of the visualizations and details of the implementation process.
   2.453 -The reason is that visualizations are based on units, constraints, and causality, all of which exist in every language (even though some older ones obfuscate units and constraints on them).
   2.454 -
   2.455 -Another benefit evident from the details in this section is that the instrumentation is implemented only once, for a language. All applications written in the language inherit the visualizations, without  change to the application code.
   2.456 -
   2.457 -\subsection{Meta-units and unit life-line in the computation model}
   2.458 -\label{subsec:LifeLine}
   2.459 -In preparation for mapping the model onto implementation details, we define a meta-unit and unit life-line. These form the basis for deciding points in the runtime  where data is collected.
   2.460 -
   2.461 -Every unit has a meta-unit that represents it in the runtime. A  unit is defined as the work that exists after leaving the runtime, up until re-entering it. For example, the trace of instructions on a core, from the point of leaving the runtime up until the next invocation. Looking at this in more detail, every runtime has some form of internal bookkeeping state for a unit, used to track constraints on it and make decisions about when and where to execute. This exists even if that state is just a pointer to a function that sits in a queue. We call this bookkeeping state for a unit the meta-unit.
   2.462 -
   2.463 -Each  unit also has a life-line, which progresses so:  creation of the meta-unit \pointer , state updates that affect constraints on the unit \pointer,   the decision is made to animate the unit  \pointer, movement of the meta-unit plus data to physical resources that do the animation \pointer  , animation of the unit, which does the work \pointer,  communication of state-update, that unit has completed, and hardware is free \pointer ,  constraint updates within runtime, possibly causing new meta-unit creations or freeing other meta-units to be chosen for animation.  This repeats for each unit. Each step is part of the model.
   2.464 -
   2.465 - Note a few implications: first, many activities internal to the runtime are part of a unit's life-line, and take place when only the meta-unit exists, before or after the work of the actual unit; second, communication that is internal to the runtime is part of the unit life-line, such as state updates; third, creation may be implied, such as in pthreads, or triggered such as in dataflow, or be by explicit command such as in StarSs. Once created, a meta-unit may languish before the unit it represents is free to be animated.
   2.466 -
   2.467 -This explains why the visualizations remain largely the same across languages. The concepts of a meta-unit, a unit, constraints on a unit, and a unit life-line are all valid in every language.  The visualizations are based on these concepts, and so likewise largely remain the same.  In the UCC, only the constraint patterns that represent  the language's constructs change between languages. In the SCG, only which construct a line in the SCG represents changes.
   2.468 -
   2.469 -\subsection{Mapping model onto implementation details in runtime}
   2.470 -
   2.471 -The model's concepts of meta-unit and unit life-line  map directly to the UCC visualization. The constraints in the UCC visualization are those stated in or implied by the application (with the complexities about UCC modifications and levels noted in Section \ref{sec:theory}). 
   2.472 -
   2.473 -However, the SCG is not a strict expression of the model, rather it's purpose is practical. It shows usage of cores, and relates that to the quantities in the model. Hence, the measurements for the SCG all are boundaries of where the core's time switches from one category in the model to a different.
   2.474 -
   2.475 -This differs from the model in subtle ways. Most notably, the model declares segments of time where communications take place, while the SCG doesn't measure the communication time directly, rather it captures idleness of the core caused by the non-overlapped portion of that communication. Also, when calculating the critical path, the SCG only counts non-overlapped portions of runtime activity. 
   2.476 -
   2.477 -This difference stems from the SCG's focus on core usage, and assigning each idle period to a cause. The runtime's choice of units to cores is what determined the source and destination of communications, which caused the idling. Hence, idle periods due to non-overlapped communication  are consequences of the  assignment choices made by the scheduler.  This supports the name: scheduling consequence graph.  
   2.478 -
   2.479 - 
   2.480 -What must be collected during the run differs between the two types of visualization. For the UCC it is unit boundaries and the constraints related to each unit. For the SCG, the same units must  be collected, but also the time a core spends on each segment of the unit's life-line.  Also, implementation details of the runtime will cause things such as idling the core during lock acquisition to be counted towards a unit's life segment. What core activities go to which life segments changes from runtime to runtime. For example, our implementation includes idle time due to acquiring the lock on shared runtime state as part of the  life-line step of state update. 
   2.481 -
   2.482 -The SCG represents each  cause of a transition from one segment of core usage to another as an arc between regions or boxes. Such a causation is always a causal dependency of some kind, because the SCG only represents physical events, even if it corresponds to a complex construct in the application. These causations are collected and  tied to one of: construct constraint, runtime internal constraint (such as must acquire lock), or hardware constraint (such as only one activity at a time on a core). In this paper, all are collected, but the only causations visualized are constructs  that cross cores, with propendent on one core and its dependent on another.
   2.483 -
   2.484 -\subsection{Instrumenting our implementation of SSR on top of VMS}
   2.485 -
   2.486 -We instrumented a version of SSR implemented on top of a proto-runtime system called VMS \cite{VMS_LCPC_11}.  This proto-runtime embodies most of a runtime implementation, but has replaced two key portions with interfaces. Those portions are the handling of language construct-constraints and the decision of which core to assign a unit to. To implement a language, one simply supplies those two portions of code, via the interface.
   2.487 -
   2.488 -VMS also has the advantage for our approach of being written in accordance with the computation model, which makes instrumenting it especially convenient. Each language construct has its own handler into which to insert measurement code, and transitions in unit life-lines also have convenient locations in VMS to insert instrumentation code.
   2.489 -
   2.490 -\subsubsection{SSR background}
   2.491 -A distinction important to understanding SSR and other parallel languages is being task-based versus virtual processor (VP) based. Task-based languages include dataflow, CnC, and StarSs.  These tasks don't suspend and resume, but rather execute to completion. Hence, such a task is the same as our definition of unit. They have no state that persists across calls to the runtime. In contrast, a virtual processor does suspend and resume and so has state that persists across runtime calls. Examples include pthreads, OpenMP thread-based constructs, UPC, and so on.
   2.492 -
   2.493 -SSR is based on virtual processors. They execute sequential code that occasionally calls a parallel construct, which suspends the VP and switches to the runtime. This means that each VP contains a sequence of units, with each unit the trace-segment between two SSR library calls.
   2.494 - 
   2.495 -
   2.496 -SSR has both deterministic constraints, which specify the source and destination VP, such as send\_from\_to, and non-de\-ter\-mi\-nis\-tic ones, in which the runtime is what chooses which VPs interact, such as send\_of\_type\_to\ and singleton. Deterministic ones display the same in the UCC and the SCG. However, non-deterministic ones need all possibilities to be determined for the UCC, requiring extra instrumentation code.
   2.497 -
   2.498 -\subsubsection{Collecting a unit}
   2.499 -Code to record a new unit is inserted into VMS at the transition out of the runtime and into application code. Code to record the unit end is inserted into the VMS primitive that performs switching back to the runtime. 
   2.500 -
   2.501 -\subsubsection{Collecting the constraints} In VMS, each language construct has its own handler. Code is inserted into each handler, to record which unit invoked the construct, and any units freed by it. The SCG links the unit that made a construct call to the units freed by that call.
   2.502 -
   2.503 -What information needs to be collected for SCG and UCC and how it is done depends on the construct:
   2.504 -\begin{itemize}
   2.505 -\item create\_VP:  We place code into the create\_VP\ handler, which records  the calling VP\ + unit, along with the newly created unit, and the VP it is assigned to. Both the SCG and UCC draw arcs between creating unit and created.
   2.506 -\item send\_from\_to and receive\_from\_to: \textbf{} Code is placed into both handlers at the point that checks if both the rendez-vous requests are present. When true, it records the two units + VPs that connected. The UCC  and SCG both represent this by two crossing dependencies.
   2.507 -\item Send\_to\_of\_type and receive\_to\_of\_type:   The same code is inserted to record  the  unit+VPs that connected. This is enough for the SCG.  But for the UCC, we want to capture all sending and receiving permutations available, so we add code that collects the group of senders and the corresponding group of receivers.
   2.508 -\item Singleton:   The singleton unit has a group of predecessor units and a group of successor units. The first predecessor to complete enables the singleton unit, while all successors must wait for its completion. We insert code into the handler, which records the predecessor that enabled the singleton unit. This is all that the SCG needs. For the UCC, we add code inside the singleton call that collects the calling unit, adding it to the predecessor group, and the unit it jumps to, adding that to the successor group.
   2.509 - 
   2.510 -\end{itemize}
   2.511 -
   2.512 -\subsubsection{Recording time, instructions, and cache misses }
   2.513 - Just recording the units and connections between them is not enough for the SCG. Because it represents core usage, the SCG also needs  the cycles spent on each activity to be collected, including internal runtime activities. The size of each interval of core usage is recorded and assigned to a  segment of some  unit's life-line.
   2.514 -
   2.515 -The UCC also makes use  of the number of instructions in a unit, as an estimate of size of work in the unit, as illustrated by Fig. \ref{fig:UCC_expl}. Without knowing the relative size of the units, it is hard to estimate the amount of parallelism \emph{usefully} available in the application.
   2.516 -
   2.517 -To measure the instructions, cycles, and communication (cache misses), we use hardware performance counters. Readings are inserted into the runtime code to capture core time spent on each segment of the life-line of a unit: 
   2.518 -\begin{enumerate}
   2.519 -\item Create meta-unit: This is the time spent inside the create\_VP construct's handler function. 
   2.520 -\item Update constraints: This is the time spent inside the handler functions that implement the constructs.
   2.521 -\item Decision to animate: This is the time spent inside the language-supplied assigner function.
   2.522 -\item Move meta-unit to core: This is via shared variables, recorded as part of 3.
   2.523 -\item Move work data to core: This is via cache misses, recorded as part of 6.
   2.524 -\item Do the work of the unit: This is the cycles between the switch-to-unit  and the following switch-to-runtime. 
   2.525 -\item Communicate state update: This is the time between leaving the application code and starting the construct handler (which includes lock acquisition).
   2.526 -\item Resulting constraint updates: This is the time spent inside the construct handler, and is the same as 2.
   2.527 -\end{enumerate}
   2.528 -
   2.529 -To cover each of those segments of a unit's life-line, code to read  the performance counters is inserted at:
   2.530 -
   2.531 -\begin{itemize}
   2.532 -\item Construct handler: To measure 2 and 8, reading is done before and after VMS calls the language-supplied construct handler function.\item
   2.533 -Assigner: To measure 3 and 4, reading is done before and after VMS calls  the language-supplied assigner function.
   2.534 -\item Work: To measure 5 and 6, reading is done by reading inside the VMS switch-to-unit operation, and the switch-to-runtime operation.
   2.535 -
   2.536 -\item 
   2.537 -Dual-use:  1 is measured by using the construct handler reads for the create\_VP construct handler. For 7, the switch-to-runtime read is subtracted from  the read at the start of the construct handler function.
   2.538 -\end{itemize}
   2.539 -
   2.540 -
   2.541 -For clarity, all but work are grouped as overhead in the visualization, but they could be displayed separately if needed.
   2.542 -
   2.543 - All the measurements are output into a trace file, which is then evaluated after the run to build the visualizations.
   2.544 -
   2.545 -\subsection{Building the Visualizations}
   2.546 -
   2.547 -Both the UCC and the SCG are internally represented as directed graphs, with units as nodes.
   2.548 -
   2.549 -\subsubsection{UCC}
   2.550 -For the UCC, units can be either unweighted or weighted. Weighted units appear as rectangles with height proportional to the weight, unweighted units appear as circles. We weight the units with the number of instructions in the work. This is not entirely execution-independent, but removes some of the influence of scheduling and data, such as cache misses.
   2.551 -
   2.552 -A critical path algorithm calculates vertical position of a unit, as its position within the critical path. The nodes are spread horizontally such that none overlap. 
   2.553 -Simple constraints (dependencies) are painted as arcs. Complicated constraints are for now displayed as an additional node bearing information on the constraint, with incoming arcs from all units whose execution status affects the constraint, and outgoing arcs going to the constrained units.
   2.554 -
   2.555 -
   2.556 -\subsubsection{SCG}
   2.557 -
   2.558 -For the SCG, all nodes are weighted with the number of cycles spent on the unit in total (work + overhead). For display, the nodes  are  split into overhead and work. The same critical path algorithm as for the UCC is used to place nodes vertically, but this time horizontal placement is determined by the core on which the unit was executed  (hardware dependencies ensure no overlap).
   2.559 -
   2.560 -Constraints can then be overlaid, color coded by type. By default, in SSR, we display creation, direct and typed message sending (but not the crossing dependency from the receiver back to the sender), and singleton outgoing dependencies; but each type can be individually hidden or shown.
   2.561 -
   2.562 -All this information is taken purely from the runtime, leading to a rich, configurable visualization without needing to add anything to the application.
   2.563 -
   2.564 -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   2.565 -\section{Conclusion}
   2.566 -\label{sec:conclusion}
   2.567 -
   2.568 -We have shown how to apply a generalized model of parallel computation to build adaptable performance visualizations, relying only on information collected through instrumenting the language runtime, with no modification to the application.
   2.569 -The approach is demonstrated through the case study of instrumenting the SSR message-passing language runtime and using it to tune a simple parallel matrix multiply.
   2.570 -
   2.571 -<<<<<<< local
   2.572 -A case study related the usability improvements in ease of identifying instances of performance loss (via the Scheduling Consequence Graph, SCG); verifying that better performance is possible (via parallelism available in the Unit and Constraint Collection, UCC); connecting  loss to the causes (via SCG) no matter if cause is application structure,  runtime implementation, or hardware feature; and  suggesting changes to the application to improve  performance.
   2.573 -=======
   2.574 -The resulting visualizations show that the focus on the parallelism-relevant concepts of work units and constraints on their execution allows a clearer view of parallelism-specific issues.
   2.575 -By integrating visual display of constraints stemming from application structure, language runtime implementation, or hardware features, the various possible causes for performance loss are covered. A flexible filtering system for different types of constraints avoids overcharging the display.
   2.576 ->>>>>>> other
   2.577 -
   2.578 -As the approach relies on information available to the runtime, we expect that even better results will be observed for ``high-level'' parallel languages that more closely match application concepts instead of hardware concepts.
   2.579 -
   2.580 -\bibliography{bib_for_papers_12_Jy_15}
   2.581 -
   2.582 -\end{document} 
   2.583 -
   2.584 -===============  
   2.585 -
   2.586 -(This is not true:) plus each point that state update directly affects    the life-state of the unit. Such state updates include life-state of other units involved in constraints with the target unit (because they may cause a change in target unit's life-state), updates of hardware state that directly cause a change in target unit's life state (such as hardware becoming free coupled to the scheduler choosing the target unit), and lastly any internal runtime state changes related to the unit (such as idling until a lock is acquired). The language provides construct-based constraints, while the runtime and hardware provide additional ones.
   2.587 -
   2.588 -===============?
   2.589 -
   2.590 -\subsection{Elements}
   2.591 -The  process of instrumenting is stra 
   2.592 -
   2.593 -?
   2.594 -
   2.595 --] Follow details up to visualization
   2.596 -
   2.597 -
   2.598 -
   2.599 - 
   2.600 --] Highlight Benefit: Main effort is just inserting collection points into runtime, and if using VMS, this is especially simple and straight-forward.
   2.601 -
   2.602 --] Effort for implementing in new language: inserting collection points into runtime, and modifying portions of CG script that connect the blocks, and modifying portions of UCC script that draw lines connecting units.
   2.603 -
   2.604 -\subsection{Elements}
   2.605 -
   2.606 -The main elements of the approach are: collection points, the output from them, and the post-processing that draws the visualizations.
   2.607 -
   2.608 -\subsubsection{Collection points}
   2.609 -
   2.610 -
   2.611 -There are two distinct kinds of collection points: those for the UCC and those for the CG.?
   2.612 -
   2.613 -The UCC requires 
   2.614 -
   2.615 -?
   2.616 -
   2.617 -
   2.618 -Collection Points:  -- this is for bottom-up and for skeptic, plus implementer
   2.619 -
   2.620 -
   2.621 --] For UCC, these are points in life-line of unit being executed where collect
   2.622 -
   2.623 --] For CG, these are points in life-line of unit being executed where collect
   2.624 -
   2.625 --] For CG, these are the points inside runtime that get collected
   2.626 -
   2.627 -
   2.628 -
   2.629 -
   2.630 -
   2.631 --] for SSR, for CG, put this collection at this point <blah blah>.. again, see VMS for more on the points..
   2.632 -
   2.633 -?
   2.634 -
   2.635 -
   2.636 -
   2.637 -Output format:   for implementer
   2.638 -
   2.639 --] The collection is made into a ListOfArrays, and dumped out in-order into a file
   2.640 -
   2.641 --] Each point in file has: <info>
   2.642 -
   2.643 -Scripts:     for implementer
   2.644 -
   2.645 --] One that does X
   2.646 -
   2.647 --] one generates SVG of UCC
   2.648 -
   2.649 ---] arranged so:
   2.650 -
   2.651 --] one generates SVG of CG
   2.652 -
   2.653 -To adapt to a new language:    for implementer
   2.654 -
   2.655 --] If use VMS, then only collection points that change are in request handlers
   2.656 -
   2.657 --] portion of script to modify is X
   2.658 -
   2.659 --] script has X lines, only Y are specific to a language
   2.660 -
   2.661 -
   2.662 -Notice, all work is inside runtime and post-processing scripts
   2.663 -
   2.664 -Notice, the process for a new language is the same, indep of particular exe model or constructs of the language -- it comes down to the same points on the lifeline of a unit, all that changes is the pattern of constraints implied by a particular construct.
   2.665 -
   2.666 -
   2.667 -
   2.668 -==================
   2.669 -
   2.670 -The visualization relies on data collected from the runtime during execution. There are two kinds of information that need to be recorded: identification of units and constraints, and execution metric measurements.
   2.671 -The first can be obtained from the language runtime at the places where constraints are checked and modified and units are created. The second have to be recorded as the unit progresses through different stages of execution.
   2.672 -
   2.673 -\subsection{Units and Constraints}
   2.674 -
   2.675 -As units are defined by scheduling decisions, the creation of a unit is easiest to register at the point where the unit is assigned to a processing element. This ensures that all units that are executed are recorded, and conversely, all units that are recorded are really executed. 
   2.676 -
   2.677 -There is no significant variation in this between languages, and the units are the same for the concrete UCC and the SCG. 
   2.678 -Language constructs specify constraints on units, where the connections between units can be very complex. Hence the instrumentation needs to be tailored to the constructs.
   2.679 -
   2.680 -Using the concrete case of SSR, we show the correspondence between the language's constructs and the elements of the UCC and the SCG.
   2.681 -
   2.682 -SSR, like threads, is based on the concept of virtual processors (VPs). These VPs execute sequential code that occasionally makes use of parallel language constructs, which suspend the VP and switch to the runtime.
   2.683 -
   2.684 - The runtime checks the constraints defined by the language, performs some operations (e.g. transmitting a message) if required, and then calls the scheduler to decide upon the next VP to be run. With our definition of units as bounded by scheduling decisions, this means that each VP contains a sequence of units, with each unit the trace-segment between two SSR library calls. 
   2.685 -
   2.686 -There is always a ``dependency'' (constraint that only frees the dependent after the propendent finished) between two successive tasks inside a same VP, since they are part of a continuous sequential code. This is a deterministic constraint, so it is the same in the UCC and the SCG. For non-deterministic constraints, such as singleton and send\_of\_type, all possibilities need to be added to the UCC. But only the actually realized choice is used in the SCG.
   2.687 -
   2.688 -Additional constraints can be placed on the tasks depending on the construct:
   2.689 -\begin{itemize}
   2.690 -\item Create VP: The creation of a new VP creates a simple dependency: the first task in the new VP may only execute after the creating task has finished.
   2.691 -\item Simple send and receive: Send to and receive from a specific VP is rendezvous based, so that the units following the communication in both VPs can only execute after the units preceding the rendezvous point in both VPs have finished. This can easily be represented by two crossing dependencies. These are deterministic, so the record is the same for the UCC or the SCG.
   2.692 -\item Typed send and receive: Typed send/receive is also rendezvous based, but contrary to simple send/receive, the pairing of sender and receiver is not deterministic. For the SCG, which represents a specific run, the actual communications observed can be recorded in the same way as simple send/receive, but for the UCC, we want to capture all sending and receiving permutations available. In this case, since the construct specifies no further constraints beyond the type of the message, we simply record for each type a group of senders and a group of receivers.
   2.693 -\item Singleton: For the singleton construct, there is a group of predecessors, the completion of one of which is sufficient to fulfil the constraint on the singleton unit. All successors have to wait for the completion of the singleton, so these are simple dependencies.
   2.694 -\end{itemize}
   2.695 -
   2.696 -\subsection{Execution measurements}
   2.697 -
   2.698 -In addition to identifying the units and the relations between them, we need to capture how much work a unit represents and how many resources (mainly core-time) it occupies.
   2.699 -
   2.700 -For performance tuning it is obviously critical to know the actual performance, but even for the UCC these measurements are important. Indeed, as evidenced by Fig. [fig:UCC_expl], without knowing the relative size of the units, it is hard so estimate the amount of parallelism in the application.
   2.701 -
   2.702 -We use hardware performance counters to measure for each unit the work (=time spent in application code) and the overhead (=time spent in runtime) in cycles and in instructions, as well as the number of last level cache misses during the work. The counter values are retrieved in the runtime before and after several sections in the lifetime of a unit: 
   2.703 -\begin{itemize}
   2.704 -\item Assign: Deciding which unit to execute next
   2.705 -\item Work: Doing the work of the unit
   2.706 -\item Invocation: Switching back from the work to the runtime
   2.707 -\item Construct handler: Performing actions requested by the application and  updating constraint status
   2.708 -\end{itemize}
   2.709 -For clarity, all but work are grouped as overhead in the visualization, but they could be displayed separately if needed.
   2.710 -
   2.711 -We also record which core a unit was executed on. All these measurements are directly output into a trace file, which is then evaluated after the run to build the visualizations.
   2.712 -
   2.713 -\subsection{Building the Visualizations}
   2.714 -
   2.715 -Both the UCC and the SCG are represented as directed graphs, with units as nodes.
   2.716 -
   2.717 -\subsubsection{UCC}
   2.718 -For the UCC, units can be either unweighted or weighted, if an estimation of the amount of work is available. Weighted units appear as rectangles with height proportional to the weight, unweighted units are circles. Our implementation can use the number of instructions in the work section from a run to weigh the units. This removes some of the influence of scheduling and data, such as cache misses, but can be insufficient if the application is strongly data- or scheduling-dependent.
   2.719 -
   2.720 -Simple, deterministic dependencies are represented as arcs. Complicated constraints are for now displayed as an additional node bearing information on the constraint with incoming arcs from all units whose execution status affects the constraint and outgoing arcs to the constrained units.
   2.721 -
   2.722 -A critical path algorithm is then used to place the nodes vertically, from top to bottom. For non-deterministic constraints, it is possible to enable or disable their participation in the path. Enabling them will lead to an over-estimation of the critical path, disabling them to an under-estimation. (Better solutions welcome) Horizontally, it is simply ensured that no nodes overlap.
   2.723 -
   2.724 -\subsubsection{SCG}
   2.725 -
   2.726 -For the SCG, all nodes are weighted with the number of cycles spent on the unit in total (work + overhead). Nodes are then further displayed separated into overhead and work. Because it displays a concrete run, the actual choices made for all non-deterministic constraints are available, so all dependencies are deterministic. The same critical path algorithm as for the UCC is used to place nodes vertically, but this time horizontal placement is determined by the core on which the unit was executed  (hardware dependencies ensure no overlap).
   2.727 -
   2.728 -A selection of constraints can then be overlaid, with color codes by type. In SSR we display by default creation, direct and typed message sending (but not the crossing dependency from the receiver to the sender), and singleton outgoing dependencies; but each type can be individually hidden or shown.
   2.729 -
   2.730 -All this information is taken purely from the runtime, leading to a rich, configurable visualization without needing to add anything to the application.
   2.731 -
   2.732 -\section{Conclusion}
   2.733 -\label{conclusion}
   2.734 -We have shown how to apply a computation model to instrument a language runtime for collecting measurements that connect: each measurement to others, to application structure, to scheduling decisions, and to hardware. A simple visualization of the data has features that indicate lost performance, and features that visually link the lost performance to the cause, no matter if the cause is application structure, language runtime implementation, or hardware feature.  It is this linkage, due to the computation model, that sets this approach apart from others. 
   2.735 - 
   2.736 -
   2.737 -\bibliography{BibForPapers}
   2.738 - 
   2.739 -\end{document}
   2.740 -
   2.741 -======
   2.742 -
   2.743 -Check-list -- be sure catch these:
   2.744 -
   2.745 -Pic illustrates matrix mult divider algorithm. 
   2.746 -
   2.747 -Pic illustrate processors created and communication between them.
   2.748 -
   2.749 -How consequence graph features each tie back to features in UCC and to specific segments of code or constructs.
   2.750 -
   2.751 -
   2.752 -Which lang features support model -- units, plus clear constraints on them -- can still do with pthreads, but less effective because don't have clean units nor semantics of constraints on units -- SSR provides these, as do languages like CnC, StarSs, and so on
   2.753 - 
   2.754 -No application instrumentation -- everything is inside language runtime (part in plugin, part in VMS -- VMS is just a helper to simplify runtime creation).
   2.755 -
   2.756 -Pic from VMS paper, with arrows pointing to places instrumentation injected.
   2.757 -
   2.758 -Avoid word "tool" in connection with visualization.. no need to draw attention to fact that we don't have a GUI, the contribution is not a tool, but rather a methodology, and the visualizations are just one element of the methodology
   2.759 -
   2.760 -Semantic information from application-code: The units, constraints on them, type of constraint, code executed in unit
   2.761 -
   2.762 -visual link perf-loss to code, as cause (or perf-loss to runtime or HW)
   2.763 -
   2.764 -visual: units, sched decision on each, hardware assigned to, runtime overhead on each, constraints bounded choice of hardware and order
   2.765 -
   2.766 -constraints color-coded by language-defined type
   2.767 -
   2.768 -visually, performance issue is pattern of either idle time or overhead
   2.769 -Together: visual links to units involved in perf-loss, which links to code-snippet, and to parallelism-construct defining the unit boundary and the construct constraining its scheduling.
   2.770 -
   2.771 -collect info by instrumenting runtime of lang -- do once, works all apps
   2.772 -
   2.773 -ties visual features to app-code segments -- despite app-independent
   2.774 -
   2.775 -Benefit: simple see perf-loss features, simple to link to cause in code, straight-forward to devise a fix.
   2.776 -
   2.777 -visual show unit linked to overhead in runtime to handle the constraints on unit, and choose its assignment to HW.
   2.778 -
   2.779 -Visual: idle of core from comm, NOT comm itself -- Rather than indicate how much time a communication takes, the visualization instead indicates how much idle time is caused on a core as a consequence of that communication.  This is the most useful-to-observe cost, of the communication. It naturally accounts for overlap, and only alerts the person tuning when there is a true cost, when the core idles.
   2.780 -
   2.781 -visual: non-ov comm + work, accounts cache-misses on out-of-order.  The second aspect of communication is the emergent effect of cache misses on computation rate.  As such, the relative size of work-units is visualized, which includes the cost of cache misses, given a shared-memory multi-core machine.
   2.782 -
   2.783 -three prevalent classes of performance loss are clearly displayed by our visualization: runtime overhead (including lock acquisition time), communication cost (both cache misses and runtime internal control communications), and poor scheduling choices (assign work in an order that causes idle time on cores)
   2.784 -
   2.785 -VMS == easy instrument lang -- Properties of VMS make it easy to instrument other languages, so the technique is straight-forward to apply to any VMS-based language.  For UCC, have to make own data struct for lang-specific dependencies.  For units, can be generic and pulled down into VMS.  VMS provides support for the instrumentation -- describe standard things support. Reuse code btwn plugins, for units, and common communication constructs (?)
   2.786 -
   2.787 - 
   2.788 -UCC is application-derived degrees of freedom for picking assignment of work to HW resources.
   2.789 -
   2.790 -N to N construct -- ex of not-dependency but still constraint
   2.791 -
   2.792 -how features indicate performance loss, and link picture elements that indicate perf loss to sources of loss -- could be param choices, like how many pieces to make, or input choices like size of matrix, or code choices, like how to perform division, or scheduling choices, like how to assign work-units to cores.
   2.793 -
   2.794 -show UCC and consqG pics..  point out glaring visual feature that says a big perf loss..  show how links to cause.. and so on..
   2.795 -
   2.796 -==============================
   2.797 -
   2.798 -\section{Random Early Thoughts}
   2.799 -
   2.800 -
   2.801 -Key places in plugin code for instrumentation -- places in the request handler that change state of a VP -- track down in plugin code every place that VP state changes, and put instrumentation there.
   2.802 -
   2.803 -If have a mutex -- is an acquire and a release -- VP only interacts with sem env, not another VP, but dependencies are between VPs, so have to record the sequence of interactions with semantic environment to establish causality chain (even though it's a runtime causilty chain, not necessarily an app constraint)
   2.804 -
   2.805 -For Vthread, no UCC, only consequence graph.
   2.806 -
   2.807 -Can apply this model to pthreads, but UCC has no useful info because units of work are not well defined -- they are instead defined in the application code -- application normally implementes an application-specific runtime as part of app -- mutexes and cond variables force adding application-related work-unit semantics to the application, in effect creating an application-specific language, via implementing an application-specific runtime.  That semantic information is not available to the pthreads language, and so is not capturable in a pthreads derived UCC. -- picture illustrates runtime, which embodies language construct semantics -- how SSR has two levels, but pthreads has three..
   2.808 -
   2.809 -=============
   2.810 -What's broken in others:
   2.811 -
   2.812 - the   or have cache misses over time, but never have a coherent view of how application code connects to what happens where and when. 
   2.813 -
   2.814 -Can have a timeline view, and next to that execution by function and what percent by function, next to that histogram of cache misses.  But no coherent view of how to connect these things. There is information missing that connects the views. The user still has to guess about what the cause might be.
   2.815 -
   2.816 -The problem of the other vis is they don't give shape of application..
   2.817 - the fundamental parallelism related structure.
   2.818 -
   2.819 -
   2.820 -to have more theoretical underpinning and several views that connect to each other. The user needs more information, with some mental framework to 
   2.821 -
   2.822 -a lot of the time it's lists of measurements, or bar graphs, things like that -- over the whole application or by function -- forcing guessing of how it connects to -- if it tells you that this line creates a lot of level 2 cache misses, that doesn't tell you what the application is doing to cause this.. but when have whole UCC along with it, have context for the measurements -- puts the line of code into a framework -- it's necessary but not useful by itself -- it needs to be connected -- the unit information is more interesting than the line of code information -- line of code has only sequential meaning, missing scheduling connection -- need the scheduling behavior added -- need to know the unit of work that's causing problem, not the line of code -- unit provides a parallelism context, line of code does not.. unit provides an execution order and execution location, with implied communication -- line of code does not.
   2.823 -
   2.824 -
   2.825 -
   2.826 -
   2.827 -======  Related work ========
   2.828 -
   2.829 -StarSs is clearly thinking about tasks, and even some about scheduling. but limited on scheduling -- can just place task into a queue, so can sort of manipulate scheduling but view doesn't really give all the constraints -- missing the runtime overhead, and missing the idle-time consequent from non-overlapped comm. 
   2.830 -
   2.831 -The PPoPP 12 paper and its ilk are fall into same category -- measurements without model to connect measurement to cause.
   2.832 -
   2.833 --] MPI event based (Paragraph)
   2.834 -
   2.835 --] Invasive Application Instrumentation approach (Paradyn)
   2.836 -
   2.837 --] Thread-model based (Tau)
   2.838 -
   2.839 --] profiling based (VTune, PPoPP 12 paper..)
   2.840 -
   2.841 --] Task-based
   2.842 -(StarSs)
   2.843 -
   2.844 -?
   2.845 -
   2.846 --] Instrument language runtime -- all applications inherit free
   2.847 -
   2.848 --] Computation model backs it, connects effect back to cause
   2.849 -
   2.850 --] Visual features indicate which hardware is idle
   2.851 -
   2.852 --] Visual features indicate cause of idleness
   2.853 -
   2.854 --] Low overhead instrumentation
   2.855 -
   2.856 --] Task carries extra info that organizes --  connects disparate measurements -- helps connect effect back to cause
   2.857 -
   2.858 --] Multiple languages easily reuse instrumentation 
   2.859 -
   2.860 --] Visual features indicate langauge features from app-code
   2.861 -
   2.862 -?
   2.863 -
   2.864 -
   2.865 -==========
   2.866 -
   2.867 -MPI is also machine-based abstraction, that gives communication information, but doesn't have concept of constraints . Its sort of in-between..
   2.868 -
   2.869 -For communication, want two things: 1) idle-time on cores that is consequence from particular communication pattern, and in some cases 2) the energy due to the volume of communication. Both are consequence of the scheduling choices made.
   2.870 -
   2.871 -
   2.872 -
   2.873 -
   2.874 -=======================?
   2.875 -
   2.876 -Their structure information is either call graph or line of code. This is too limited for parallel performance. Need tasks and scheduling. More recent languages move towards task-based. This makes more of the needed structure information available. But we've not seen a complete or coherent presentation.
   2.877 -
   2.878 -=====
   2.879 - 
   2.880 - The value of a consequence graph is linking the size of boxes in it to the decisions made by the scheduler, which are represented by the shape. This lets a person visually link scheduling decisions to consequences.  The mind can quickly see empty areas, representing lost performance, and connect those to boxes and lines that cause the empty areas.  They then look up the code of each box and construct of each line, which gives them the code structures that caused the loss of performance.  This \emph{visual} linkage of undesirable effect back to \emph{cause in the code} is what sets our approach apart. 
   2.881 -The value of the UCC is visually linking cause of performance loss to options available for fixing it.
   2.882 -
   2.883 -===== 
   2.884 -
   2.885 -
   2.886 -
   2.887 -A survey of the most highly cited classic papers shows the commonality..
   2.888 -
   2.889 -===========
   2.890 -
   2.891 -
   2.892 -Use case: takes longer to create when more units -- that's because it mallocs all the matrix pieces, before it pairs any and starts the pair as work -- can see this from the graphs by linking the units to code they execute -- not the functions, specifically, but to the scheduled code-snippet. -- suggests changing the way divider works.
   2.893 -
   2.894 -Graph tells us the division code is bad -- works better when have 39 cores instead of 40 -- makes more pieces for 39 than for 40, and the end up fitting better onto the cores..
   2.895 -
   2.896 -Systematic way of adding auto-tuning to language -- using TSC for time measurements, and counters for cache miss.  Provides a framework that links the pieces together, and the UCC gives critical information useful in tuning.
   2.897 -
   2.898 -=======================
   2.899 -
   2.900 -Advantage: unlike statistical, it catches all "one off" and momentary serializations, which cause massive idling.
   2.901 -
   2.902 -Advantage: it is quick and easy to instrument a language that uses the same runtime abstraction (VMS) -- VMS was made to make language runtimes quick and easy, with low overhead (cite). VMS has the base instrumentation in it, language adds the types of constraints and the collection points. Each request to the runtime is in a separate (inlined) function, which makes adding the instrumentation simple.
   2.903 -
   2.904 -Advantage: instrument the language, NOT the application -- and yet it gives all the application-specific information from a run. -- others: Tau forces extensive instrumentation of application..  MPI based ones don't give the constraint info, nor what occupies time of core info..
   2.905 -
   2.906 -=======================
   2.907 -   
   2.908 -Expl of what is meant by "structure" info -- example where meas of runtime system showed that overhead of task creation took longer than task execution. Hypothesis was trivial: cause of lost performance is runtime overhead of creation is larger than work in a scheduled unit.  The plan to fix is to change the number of work-units created, by changing the parameter in the divider code.  Implementing this and re-executing showed that this source of performance loss was fixed by the change.
   2.909 -
   2.910 -The example shows that theory is part of hypothesis generation, because it required knowledge of the runtime and understanding that creation of a task is work performed as overhead inside the runtime.  The example also shows that generating the plan to fix required understanding the segment of code that divided work into tasks, and the relationship between parameters to that code and execution time of the resulting tasks.
   2.911 -
   2.912 -Hence, each step of performance debugging involves several aspects, including mental model of computation, application code, runtime implementation, scheduling choices, and hardware. In order to be effective, a tool used during performance tuning must be part of a complete model of computation that ties all aspects of the debugging/tuning steps together. Current tools fall short, both because they lack an encompassing model of computation, and because the tools are isolated from each other. Without integration, the user gets an incomplete picture of the computation and must resort to guesses either of where the problem lies or of what to do to fix it.
   2.913 -
   2.914 -
   2.915 -We introduce in this paper a model of computation that ties all aspects of performance together, along with instrumentation and visualization that is guided by the model and that links all relevant performance tuning information together. The model and visualization tools are illustrated with a case study, which shows how they are used to performance tune the standard matrix-multiply application on two multi-core systems. 
   2.916 -
   2.917 -Although we use standard visualization techniques [cite], our approach differs from previous work in both theoretical and practical aspects. The theory we  use is The Holistic Model of Parallel Computation, which ties together parallelism construct semantics with scheduling choices made during a run, and specific measurements made on the cores.  When put into practice, new kinds of measurements are taken, which complete the picture presented to the user, and each measurement is tied to a specific segment of code. The resulting combination not only identifies each source of performance loss, but ties it back to specific causes and suggests precise fixes, all of which is illustrated in our story line.
   2.918 -
   2.919 -[maybe some stuff about features and benefits of our approach: no app instrumentation, it's all inside language runtime, very low overhead, integrated with VMS-based functional debugging, and so on]
   2.920 -
   2.921 -
   2.922 -Table of features/benefits comparing us to others -- in related work section.
   2.923 --- language invariant.. concepts are lang independent, visualization is language independent
   2.924 --- instr lang once, then works across apps
   2.925 --- Integrate measurements with runtime implementation details
   2.926 --- Integrate measurements of one kind with those of a different kind
   2.927 --- Integrate measurements of core activity with code snippets
   2.928 --- Underlying theory ties measurements to each other, to runtime, and to hardware
   2.929 --- Has concept of unit of work / task
   2.930 --- Includes communication between cores
   2.931 --- Shows what each core spends each bit of time on
   2.932 --- Visual representation directly indicates areas of performance loss
   2.933 -
   2.934 -\begin{table*}[ht!]
   2.935 -\caption
   2.936 -{Comparison of Features and Benefits
   2.937 -}
   2.938 -\label{tabPersonDaysLang}
   2.939 -\begin{center}
   2.940 -\begin{tabular}{llll}
   2.941 -           & Tau & Paradyn & ASH \\
   2.942 - Has Underlying Parallel Computation Model &  &  & Y\\
   2.943 - Has concept of schedulable unit of work &  &  & Y\\
   2.944 - Visualization is Invariant Across Languages      &  &  & Y\\
   2.945 - Instrument Language only (not app code) &  &  & Y\\
   2.946 - Integrates Measurements into whole picture &  &  & Y\\
   2.947 - Visualizes all core activities &  &  & Y\\
   2.948 - Core's activities are integrated with scheduling decisions &  &  & Y\\
   2.949 - Visualizes runtime overhead &  &  & Y\\
   2.950 - Visualizes consequences of communications &  &  & Y\\
   2.951 - Visually suggests \emph{hypothesis} for performance loss &  &  & Y\\
   2.952 -  &  &  & Y\\
   2.953 -  &  &  & Y\\
   2.954 -  &  &  & Y\\
   2.955 -      L.O.C. &     470 & 290 & 310
   2.956 -  \end{tabular}
   2.957 -\end{center}
   2.958 -\end{table*}
   2.959 -
   2.960 -
   2.961 -===================
   2.962 -
   2.963 -Perf bounded by: the application-imposed constraints on scheduling units, the hardware-imposed constraints on scheduling them, and the runtime-implementation imposed constraints on scheduling them. 
   2.964 -
   2.965 -
   2.966 -Par Comp Model:  Scheduling is a fundamental part of parallel execution. The views must include both constraints on scheduling and the actual scheduling choices. The parts that affect what scheduling choices are possible must connect to the parts that show which ones were taken.
   2.967 -
   2.968 -
   2.969 -==========  Table ref, figure environment, table environment
   2.970 - \tabref{table1}.
   2.971 - 
   2.972 -Using the Figure environment:
   2.973 -\begin{Figure}[h]{figurelabel}% <- don't forget this percent!
   2.974 -        {Caption for figure, Figure environment.}
   2.975 -\fbox{The figure matter goes here.}
   2.976 -\end{Figure}
   2.977 -
   2.978 -Table environment:
   2.979 -\begin{Table}{tablelabel}% <- don't forget this percent!
   2.980 -        {Caption for table, Table environment.}
   2.981 -\fbox{The table matter goes here.}
   2.982 -\end{Table}
   2.983 -
   2.984 -
   2.985 -%%
   2.986 -%% EOF ieeepes_skel.tex
   2.987 -%%----------------------------------------------------------------------
   2.988 -
   2.989 - destroying virtutal processors, and three kinds of send-receive pairs. The 
   2.990 - graph. It depicts all the scheduling operations performed by the runtime, 
   2.991 -
   2.992 -
   2.993 -
   2.994 -
   2.995 - all, why go parallel, if not for higher performance? Yet when trying to track 
   2.996 -
   2.997 -
   2.998 -
   2.999 -
  2.1000 -
  2.1001 -
  2.1002 -
  2.1003 -
  2.1004 -
  2.1005 -
  2.1006 -
  2.1007 -
  2.1008 -
  2.1009 -
  2.1010 -
  2.1011 -
  2.1012 -
  2.1013 -
  2.1014 -
  2.1015 -
  2.1016 -
  2.1017 -
  2.1018 -
  2.1019 -
  2.1020 -
  2.1021 -
  2.1022 -
  2.1023 -
  2.1024 -
  2.1025 -
  2.1026 -
  2.1027 -
  2.1028 -
  2.1029 -
  2.1030 -
  2.1031 -
  2.1032 -
  2.1033 -
  2.1034 -
  2.1035 -
  2.1036 -
  2.1037 -
  2.1038 -
  2.1039 -
  2.1040 -
  2.1041 -
  2.1042 -
  2.1043 -
  2.1044 -
  2.1045 -
  2.1046 -
  2.1047 -
  2.1048 -
  2.1049 -
  2.1050 -
  2.1051 -
  2.1052 -
  2.1053 -
  2.1054 -
  2.1055 -
  2.1056 -
  2.1057 -
  2.1058 -
  2.1059 -
  2.1060 -
  2.1061 -
  2.1062 -
  2.1063 -
  2.1064 -
  2.1065 -
  2.1066 -
  2.1067 -
  2.1068 -
  2.1069 -
  2.1070 -
  2.1071 -
  2.1072 -
  2.1073 -
  2.1074 -
  2.1075 -
  2.1076 -
  2.1077 -
  2.1078 -
  2.1079 -
  2.1080 -
  2.1081 -
  2.1082 -
  2.1083 -
  2.1084 -
  2.1085 -
  2.1086 -
  2.1087 -
  2.1088 -
  2.1089 -
  2.1090 -
  2.1091 -
  2.1092 -
  2.1093 -
  2.1094 -
  2.1095 -
  2.1096 -
  2.1097 -
  2.1098 -
  2.1099 -
  2.1100 -
  2.1101 -
  2.1102 -
  2.1103 -
  2.1104 -
  2.1105 -
  2.1106 -
  2.1107 -
  2.1108 -
  2.1109 -
  2.1110 -
  2.1111 -
  2.1112 -
  2.1113 -
  2.1114 -
  2.1115 -
  2.1116 -
  2.1117 -
  2.1118 -
  2.1119 -
  2.1120 -
  2.1121 -
  2.1122 -
  2.1123 -
  2.1124 -
  2.1125 -
  2.1126 -
  2.1127 -
  2.1128 -
  2.1129 -
  2.1130 -
  2.1131 -
  2.1132 -
  2.1133 -
  2.1134 -
  2.1135 -
  2.1136 -
  2.1137 -
  2.1138 -
  2.1139 -
  2.1140 -
  2.1141 -
  2.1142 -
  2.1143 -
  2.1144 -
  2.1145 -
  2.1146 -
  2.1147 -
  2.1148 -
  2.1149 -
  2.1150 -
  2.1151 -
  2.1152 -
  2.1153 -
  2.1154 -
  2.1155 -
  2.1156 -
  2.1157 -
  2.1158 -
  2.1159 -
  2.1160 -
  2.1161 -
  2.1162 -
  2.1163 -
  2.1164 -
  2.1165 -
  2.1166 -
  2.1167 -
  2.1168 -
  2.1169 -
  2.1170 -
  2.1171 -
  2.1172 -
  2.1173 -
  2.1174 -
  2.1175 -
  2.1176 -
  2.1177 -
  2.1178 -
  2.1179 -
  2.1180 -
  2.1181 -
  2.1182 -
  2.1183 -
  2.1184 -
  2.1185 -
  2.1186 -
  2.1187 -
  2.1188 -
  2.1189 -
  2.1190 -
  2.1191 -
  2.1192 -
  2.1193 -
  2.1194 -
  2.1195 -
  2.1196 -
  2.1197 -
  2.1198 -
  2.1199 -
  2.1200 -
  2.1201 -
  2.1202 -
  2.1203 -
  2.1204 -
  2.1205 -
  2.1206 -
  2.1207 -
  2.1208 -
  2.1209 -
  2.1210 -
  2.1211 -
  2.1212 -
  2.1213 -
  2.1214 -
  2.1215 -
  2.1216 -
  2.1217 -
  2.1218 -
  2.1219 -
  2.1220 -
  2.1221 -
  2.1222 -
  2.1223 -
  2.1224 -
  2.1225 -
  2.1226 -
  2.1227 -
  2.1228 -
  2.1229 -
  2.1230 -
  2.1231 -
  2.1232 -
  2.1233 -
  2.1234 -
  2.1235 -
  2.1236 -
  2.1237 -
  2.1238 -
  2.1239 -
  2.1240 -
  2.1241 -
  2.1242 -
  2.1243 -
  2.1244 -
  2.1245 -
  2.1246 -
  2.1247 -
  2.1248 -
  2.1249 -
  2.1250 -
  2.1251 - trying to pinpoint the causes of performance loss,  internal structure of 
  2.1252 -

     3.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     3.2 +++ b/0__Papers/PRT/PRT__Tutorial_explanation_teaching/figures/PR__system_level_activity.svg	Fri Sep 13 11:02:18 2013 -0700
     3.3 @@ -0,0 +1,2923 @@
     3.4 +<?xml version="1.0" encoding="UTF-8" standalone="no"?>
     3.5 +<!-- Created with Inkscape (http://www.inkscape.org/) -->
     3.6 +
     3.7 +<svg
     3.8 +   xmlns:dc="http://purl.org/dc/elements/1.1/"
     3.9 +   xmlns:cc="http://creativecommons.org/ns#"
    3.10 +   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    3.11 +   xmlns:svg="http://www.w3.org/2000/svg"
    3.12 +   xmlns="http://www.w3.org/2000/svg"
    3.13 +   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
    3.14 +   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
    3.15 +   width="744.09448819"
    3.16 +   height="1052.3622047"
    3.17 +   id="svg2"
    3.18 +   sodipodi:version="0.32"
    3.19 +   inkscape:version="0.48.2 r9819"
    3.20 +   sodipodi:docname="PR__system_level_activity.svg"
    3.21 +   inkscape:output_extension="org.inkscape.output.svg.inkscape"
    3.22 +   version="1.1">
    3.23 +  <defs
    3.24 +     id="defs4">
    3.25 +    <marker
    3.26 +       inkscape:stockid="Arrow2Mend"
    3.27 +       orient="auto"
    3.28 +       refY="0.0"
    3.29 +       refX="0.0"
    3.30 +       id="Arrow2Mend"
    3.31 +       style="overflow:visible;">
    3.32 +      <path
    3.33 +         id="path4008"
    3.34 +         style="font-size:12.0;fill-rule:evenodd;stroke-width:0.62500000;stroke-linejoin:round;"
    3.35 +         d="M 8.7185878,4.0337352 L -2.2072895,0.016013256 L 8.7185884,-4.0017078 C 6.9730900,-1.6296469 6.9831476,1.6157441 8.7185878,4.0337352 z "
    3.36 +         transform="scale(0.6) rotate(180) translate(0,0)" />
    3.37 +    </marker>
    3.38 +    <inkscape:perspective
    3.39 +       sodipodi:type="inkscape:persp3d"
    3.40 +       inkscape:vp_x="0 : 526.18109 : 1"
    3.41 +       inkscape:vp_y="0 : 1000 : 0"
    3.42 +       inkscape:vp_z="744.09448 : 526.18109 : 1"
    3.43 +       inkscape:persp3d-origin="372.04724 : 350.78739 : 1"
    3.44 +       id="perspective10" />
    3.45 +    <inkscape:perspective
    3.46 +       id="perspective12172"
    3.47 +       inkscape:persp3d-origin="0.5 : 0.33333333 : 1"
    3.48 +       inkscape:vp_z="1 : 0.5 : 1"
    3.49 +       inkscape:vp_y="0 : 1000 : 0"
    3.50 +       inkscape:vp_x="0 : 0.5 : 1"
    3.51 +       sodipodi:type="inkscape:persp3d" />
    3.52 +    <inkscape:perspective
    3.53 +       id="perspective12937"
    3.54 +       inkscape:persp3d-origin="0.5 : 0.33333333 : 1"
    3.55 +       inkscape:vp_z="1 : 0.5 : 1"
    3.56 +       inkscape:vp_y="0 : 1000 : 0"
    3.57 +       inkscape:vp_x="0 : 0.5 : 1"
    3.58 +       sodipodi:type="inkscape:persp3d" />
    3.59 +  </defs>
    3.60 +  <sodipodi:namedview
    3.61 +     id="base"
    3.62 +     pagecolor="#ffffff"
    3.63 +     bordercolor="#666666"
    3.64 +     borderopacity="1.0"
    3.65 +     gridtolerance="10000"
    3.66 +     guidetolerance="10"
    3.67 +     objecttolerance="10"
    3.68 +     inkscape:pageopacity="0.0"
    3.69 +     inkscape:pageshadow="2"
    3.70 +     inkscape:zoom="1.4897651"
    3.71 +     inkscape:cx="290.73583"
    3.72 +     inkscape:cy="795.84567"
    3.73 +     inkscape:document-units="px"
    3.74 +     inkscape:current-layer="layer1"
    3.75 +     showgrid="false"
    3.76 +     inkscape:window-width="1600"
    3.77 +     inkscape:window-height="848"
    3.78 +     inkscape:window-x="-8"
    3.79 +     inkscape:window-y="-8"
    3.80 +     inkscape:window-maximized="1" />
    3.81 +  <metadata
    3.82 +     id="metadata7">
    3.83 +    <rdf:RDF>
    3.84 +      <cc:Work
    3.85 +         rdf:about="">
    3.86 +        <dc:format>image/svg+xml</dc:format>
    3.87 +        <dc:type
    3.88 +           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
    3.89 +        <dc:title></dc:title>
    3.90 +      </cc:Work>
    3.91 +    </rdf:RDF>
    3.92 +  </metadata>
    3.93 +  <g
    3.94 +     inkscape:label="Layer 1"
    3.95 +     inkscape:groupmode="layer"
    3.96 +     id="layer1">
    3.97 +    <text
    3.98 +       sodipodi:linespacing="100%"
    3.99 +       id="text12979"
   3.100 +       y="515.76575"
   3.101 +       x="326.92542"
   3.102 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.103 +       xml:space="preserve"><tspan
   3.104 +         y="515.76575"
   3.105 +         x="326.92542"
   3.106 +         id="tspan12981"
   3.107 +         sodipodi:role="line"
   3.108 +         style="text-align:center;text-anchor:middle">Master</tspan><tspan
   3.109 +         y="524.20056"
   3.110 +         x="326.92542"
   3.111 +         sodipodi:role="line"
   3.112 +         id="tspan3291"
   3.113 +         style="font-size:8px;text-align:center;text-anchor:middle">(runtime system)</tspan><tspan
   3.114 +         y="532.2005"
   3.115 +         x="326.92542"
   3.116 +         sodipodi:role="line"
   3.117 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.118 +         id="tspan8409">on core 1</tspan></text>
   3.119 +    <g
   3.120 +       id="g9336"
   3.121 +       transform="translate(44.002789,114)">
   3.122 +      <rect
   3.123 +         style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0"
   3.124 +         id="rect3293"
   3.125 +         width="65.276405"
   3.126 +         height="37.165791"
   3.127 +         x="346.40247"
   3.128 +         y="389.58255" />
   3.129 +      <text
   3.130 +         sodipodi:linespacing="100%"
   3.131 +         id="text3295"
   3.132 +         y="400.15268"
   3.133 +         x="379.00162"
   3.134 +         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.135 +         xml:space="preserve"><tspan
   3.136 +           id="tspan3301"
   3.137 +           y="400.15268"
   3.138 +           x="379.00162"
   3.139 +           sodipodi:role="line"
   3.140 +           style="text-align:center;text-anchor:middle;fill:#008000">Seed VP</tspan><tspan
   3.141 +           y="408.58746"
   3.142 +           x="379.00162"
   3.143 +           sodipodi:role="line"
   3.144 +           style="font-size:8px;text-align:center;text-anchor:middle;fill:#008000"
   3.145 +           id="tspan9389">(created at</tspan><tspan
   3.146 +           y="416.58746"
   3.147 +           x="379.00162"
   3.148 +           sodipodi:role="line"
   3.149 +           style="font-size:8px;text-align:center;text-anchor:middle;fill:#008000"
   3.150 +           id="tspan9391">app startup,</tspan><tspan
   3.151 +           y="424.58746"
   3.152 +           x="379.00162"
   3.153 +           sodipodi:role="line"
   3.154 +           style="font-size:8px;text-align:center;text-anchor:middle;fill:#008000"
   3.155 +           id="tspan9744">on core 1)</tspan></text>
   3.156 +    </g>
   3.157 +    <text
   3.158 +       xml:space="preserve"
   3.159 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.160 +       x="517.16797"
   3.161 +       y="467.42465"
   3.162 +       id="text3318-1"
   3.163 +       sodipodi:linespacing="100%"><tspan
   3.164 +         style="text-align:center;text-anchor:middle;fill:#008000"
   3.165 +         sodipodi:role="line"
   3.166 +         x="517.16797"
   3.167 +         y="467.42465"
   3.168 +         id="tspan3322-5">Application Code</tspan></text>
   3.169 +    <text
   3.170 +       sodipodi:linespacing="100%"
   3.171 +       id="text3376"
   3.172 +       y="541.42468"
   3.173 +       x="517.16797"
   3.174 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.175 +       xml:space="preserve"><tspan
   3.176 +         id="tspan3378"
   3.177 +         y="541.42468"
   3.178 +         x="517.16797"
   3.179 +         sodipodi:role="line"
   3.180 +         style="text-align:center;text-anchor:middle;fill:#008000">Seed_Fn</tspan></text>
   3.181 +    <text
   3.182 +       xml:space="preserve"
   3.183 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.184 +       x="517.16797"
   3.185 +       y="761.42468"
   3.186 +       id="text3380"
   3.187 +       sodipodi:linespacing="100%"><tspan
   3.188 +         style="text-align:center;text-anchor:middle;fill:#008000"
   3.189 +         sodipodi:role="line"
   3.190 +         x="517.16797"
   3.191 +         y="761.42468"
   3.192 +         id="tspan3382">Work_Fn</tspan></text>
   3.193 +    <text
   3.194 +       sodipodi:linespacing="100%"
   3.195 +       id="text3388"
   3.196 +       y="467.42465"
   3.197 +       x="604.63812"
   3.198 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.199 +       xml:space="preserve"><tspan
   3.200 +         id="tspan3390"
   3.201 +         y="467.42465"
   3.202 +         x="606.14203"
   3.203 +         sodipodi:role="line"
   3.204 +         style="text-align:center;text-anchor:middle;fill:#800000">Language </tspan><tspan
   3.205 +         y="477.42465"
   3.206 +         x="604.63812"
   3.207 +         sodipodi:role="line"
   3.208 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.209 +         id="tspan5106">Wrapper-Lib</tspan><tspan
   3.210 +         y="487.42465"
   3.211 +         x="604.63812"
   3.212 +         sodipodi:role="line"
   3.213 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.214 +         id="tspan3392">Code</tspan></text>
   3.215 +    <text
   3.216 +       xml:space="preserve"
   3.217 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.218 +       x="669.16797"
   3.219 +       y="467.42465"
   3.220 +       id="text3394"
   3.221 +       sodipodi:linespacing="100%"><tspan
   3.222 +         style="text-align:center;text-anchor:middle;fill:#000080"
   3.223 +         sodipodi:role="line"
   3.224 +         x="669.16797"
   3.225 +         y="467.42465"
   3.226 +         id="tspan3396">Proto-Runtime</tspan><tspan
   3.227 +         style="text-align:center;text-anchor:middle;fill:#000080"
   3.228 +         sodipodi:role="line"
   3.229 +         x="669.16797"
   3.230 +         y="477.42465"
   3.231 +         id="tspan4998">Primitive</tspan><tspan
   3.232 +         id="tspan3398"
   3.233 +         style="text-align:center;text-anchor:middle;fill:#000080"
   3.234 +         sodipodi:role="line"
   3.235 +         x="669.16797"
   3.236 +         y="487.42465">Code</tspan></text>
   3.237 +    <text
   3.238 +       xml:space="preserve"
   3.239 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.240 +       x="137.31689"
   3.241 +       y="573.42468"
   3.242 +       id="text3412"
   3.243 +       sodipodi:linespacing="100%"><tspan
   3.244 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.245 +         sodipodi:role="line"
   3.246 +         x="137.31689"
   3.247 +         y="573.42468"
   3.248 +         id="tspan3414">Lang Handler</tspan><tspan
   3.249 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.250 +         sodipodi:role="line"
   3.251 +         x="137.31689"
   3.252 +         y="583.42468"
   3.253 +         id="tspan3422">for create VP</tspan></text>
   3.254 +    <text
   3.255 +       xml:space="preserve"
   3.256 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.257 +       x="137.68066"
   3.258 +       y="621.42468"
   3.259 +       id="text3444"
   3.260 +       sodipodi:linespacing="100%"><tspan
   3.261 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.262 +         sodipodi:role="line"
   3.263 +         x="137.68066"
   3.264 +         y="621.42468"
   3.265 +         id="tspan3450">Assigner Fn</tspan></text>
   3.266 +    <text
   3.267 +       xml:space="preserve"
   3.268 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.269 +       x="327.16797"
   3.270 +       y="469.42465"
   3.271 +       id="text3456"
   3.272 +       sodipodi:linespacing="100%"><tspan
   3.273 +         style="text-align:center;text-anchor:middle"
   3.274 +         sodipodi:role="line"
   3.275 +         x="328.67188"
   3.276 +         y="469.42465"
   3.277 +         id="tspan3458">Instances of </tspan><tspan
   3.278 +         style="text-align:center;text-anchor:middle"
   3.279 +         sodipodi:role="line"
   3.280 +         x="327.16797"
   3.281 +         y="479.42465"
   3.282 +         id="tspan8405">runtime system</tspan><tspan
   3.283 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.284 +         sodipodi:role="line"
   3.285 +         x="327.16797"
   3.286 +         y="487.85944"
   3.287 +         id="tspan9383">(data structs</tspan><tspan
   3.288 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.289 +         sodipodi:role="line"
   3.290 +         x="327.16797"
   3.291 +         y="495.85944"
   3.292 +         id="tspan9385">on heap)</tspan><tspan
   3.293 +         id="tspan3460"
   3.294 +         style="text-align:center;text-anchor:middle"
   3.295 +         sodipodi:role="line"
   3.296 +         x="327.16797"
   3.297 +         y="503.85944" /></text>
   3.298 +    <text
   3.299 +       sodipodi:linespacing="100%"
   3.300 +       id="text3462"
   3.301 +       y="467.42465"
   3.302 +       x="423.16797"
   3.303 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.304 +       xml:space="preserve"><tspan
   3.305 +         id="tspan3464"
   3.306 +         y="467.42465"
   3.307 +         x="423.16797"
   3.308 +         sodipodi:role="line"
   3.309 +         style="text-align:center;text-anchor:middle">Instances of</tspan><tspan
   3.310 +         y="477.42465"
   3.311 +         x="423.16797"
   3.312 +         sodipodi:role="line"
   3.313 +         style="text-align:center;text-anchor:middle"
   3.314 +         id="tspan3468">Virtual Processors</tspan><tspan
   3.315 +         y="485.85944"
   3.316 +         x="423.16797"
   3.317 +         sodipodi:role="line"
   3.318 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.319 +         id="tspan3466">(data structs</tspan><tspan
   3.320 +         y="493.85944"
   3.321 +         x="423.16797"
   3.322 +         sodipodi:role="line"
   3.323 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.324 +         id="tspan9387">on heap)</tspan></text>
   3.325 +    <path
   3.326 +       inkscape:connector-curvature="0"
   3.327 +       id="path4796"
   3.328 +       d="m 457.32649,538.02605 c 38.42351,0 38.42351,0 38.42351,0"
   3.329 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999996, 2.99999996;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
   3.330 +    <text
   3.331 +       xml:space="preserve"
   3.332 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.333 +       x="604.78461"
   3.334 +       y="535.42468"
   3.335 +       id="text4988"
   3.336 +       sodipodi:linespacing="100%"><tspan
   3.337 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.338 +         sodipodi:role="line"
   3.339 +         x="604.78461"
   3.340 +         y="535.42468"
   3.341 +         id="tspan4990">create VP</tspan><tspan
   3.342 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.343 +         sodipodi:role="line"
   3.344 +         x="604.78461"
   3.345 +         y="545.42468"
   3.346 +         id="tspan5104">wrapper Fn</tspan></text>
   3.347 +    <text
   3.348 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"
   3.349 +       xml:space="preserve"
   3.350 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.351 +       x="640.93384"
   3.352 +       y="566.94653"
   3.353 +       id="text5064"
   3.354 +       sodipodi:linespacing="100%"><tspan
   3.355 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.356 +         sodipodi:role="line"
   3.357 +         x="640.93384"
   3.358 +         y="566.94653"
   3.359 +         id="tspan5066">Call to dyn lib</tspan></text>
   3.360 +    <path
   3.361 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.00000003, 3.00000003;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.362 +       d="m 291.99816,538.02605 c -63.44071,0 -63.44071,0 -63.44071,0"
   3.363 +       id="path5068"
   3.364 +       inkscape:connector-curvature="0" />
   3.365 +    <text
   3.366 +       sodipodi:linespacing="100%"
   3.367 +       id="text5070"
   3.368 +       y="574.53986"
   3.369 +       x="148.45142"
   3.370 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.371 +       xml:space="preserve"
   3.372 +       transform="matrix(0.97994281,-0.19927893,0.19927893,0.97994281,0,0)"><tspan
   3.373 +         id="tspan5072"
   3.374 +         y="574.53986"
   3.375 +         x="148.45142"
   3.376 +         sodipodi:role="line"
   3.377 +         style="font-size:8px;text-align:center;text-anchor:middle">Top Level Fn</tspan></text>
   3.378 +    <text
   3.379 +       transform="matrix(0.97994281,-0.19927893,0.19927893,0.97994281,0,0)"
   3.380 +       xml:space="preserve"
   3.381 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.382 +       x="368.35757"
   3.383 +       y="617.21826"
   3.384 +       id="text5074"
   3.385 +       sodipodi:linespacing="100%"><tspan
   3.386 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.387 +         sodipodi:role="line"
   3.388 +         x="368.35757"
   3.389 +         y="617.21826"
   3.390 +         id="tspan5076">Top Level Fn</tspan></text>
   3.391 +    <text
   3.392 +       sodipodi:linespacing="100%"
   3.393 +       id="text5078"
   3.394 +       y="833.20477"
   3.395 +       x="326.47617"
   3.396 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.397 +       xml:space="preserve"
   3.398 +       transform="matrix(0.97994281,-0.19927893,0.19927893,0.97994281,0,0)"><tspan
   3.399 +         id="tspan5080"
   3.400 +         y="833.20477"
   3.401 +         x="326.47617"
   3.402 +         sodipodi:role="line"
   3.403 +         style="font-size:8px;text-align:center;text-anchor:middle">Top Level Fn</tspan></text>
   3.404 +    <path
   3.405 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999996, 2.99999996;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.406 +       d="m 458.9835,758.02605 c 36.7665,0 36.7665,0 36.7665,0"
   3.407 +       id="path5082"
   3.408 +       inkscape:connector-curvature="0" />
   3.409 +    <path
   3.410 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.411 +       d="m 514.02904,544.01894 c 0,5.6572 0,5.6572 0,5.6572"
   3.412 +       id="path5092"
   3.413 +       inkscape:connector-curvature="0"
   3.414 +       sodipodi:nodetypes="cc" />
   3.415 +    <path
   3.416 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.417 +       d="m 514.24714,550.02605 c 85.56237,0 85.56237,0 85.56237,0"
   3.418 +       id="path5094"
   3.419 +       inkscape:connector-curvature="0" />
   3.420 +    <path
   3.421 +       inkscape:connector-curvature="0"
   3.422 +       id="path5096"
   3.423 +       d="m 602.77166,658.02605 c -88.41298,0 -88.41298,0 -88.41298,0"
   3.424 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
   3.425 +    <path
   3.426 +       sodipodi:nodetypes="cc"
   3.427 +       inkscape:connector-curvature="0"
   3.428 +       id="path5140"
   3.429 +       d="m 514.02904,658.31987 c 0,26.21842 0,26.21842 0,26.21842"
   3.430 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.431 +    <path
   3.432 +       inkscape:connector-curvature="0"
   3.433 +       id="path5142"
   3.434 +       d="m 514.2471,684.02605 c 87.58783,0 87.58783,0 87.58783,0"
   3.435 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
   3.436 +    <text
   3.437 +       sodipodi:linespacing="100%"
   3.438 +       id="text5157"
   3.439 +       y="670.71179"
   3.440 +       x="604.78461"
   3.441 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.442 +       xml:space="preserve"><tspan
   3.443 +         id="tspan5161"
   3.444 +         y="670.71179"
   3.445 +         x="604.78461"
   3.446 +         sodipodi:role="line"
   3.447 +         style="text-align:center;text-anchor:middle;fill:#800000">end VP</tspan><tspan
   3.448 +         y="680.71179"
   3.449 +         x="604.78461"
   3.450 +         sodipodi:role="line"
   3.451 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.452 +         id="tspan5207">wrapper Fn</tspan></text>
   3.453 +    <path
   3.454 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.455 +       d="m 602.26584,690.02605 c 63.99676,0 63.99676,0 63.99676,0"
   3.456 +       id="path5165"
   3.457 +       inkscape:connector-curvature="0" />
   3.458 +    <path
   3.459 +       sodipodi:nodetypes="cc"
   3.460 +       inkscape:connector-curvature="0"
   3.461 +       id="path5179"
   3.462 +       d="m 514.02904,764.01895 c 0,5.6572 0,5.6572 0,5.6572"
   3.463 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.464 +    <path
   3.465 +       sodipodi:nodetypes="cc"
   3.466 +       inkscape:connector-curvature="0"
   3.467 +       id="path5209"
   3.468 +       d="m 602.02904,682.31985 c 0,7.35629 0,7.35629 0,7.35629"
   3.469 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.470 +    <text
   3.471 +       xml:space="preserve"
   3.472 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.473 +       x="497.16797"
   3.474 +       y="399.42468"
   3.475 +       id="text5211"
   3.476 +       sodipodi:linespacing="100%"><tspan
   3.477 +         style="text-align:center;text-anchor:middle"
   3.478 +         sodipodi:role="line"
   3.479 +         x="497.16797"
   3.480 +         y="399.42468"
   3.481 +         id="tspan5215" /></text>
   3.482 +    <path
   3.483 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.484 +       d="m 602.02904,550.21843 c 0,19.4577 0,19.4577 0,19.4577"
   3.485 +       id="path5217"
   3.486 +       inkscape:connector-curvature="0"
   3.487 +       sodipodi:nodetypes="cc" />
   3.488 +    <path
   3.489 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.490 +       d="m 602.24714,570.02605 c 64.71029,0 64.71029,0 64.71029,0"
   3.491 +       id="path5219"
   3.492 +       inkscape:connector-curvature="0" />
   3.493 +    <path
   3.494 +       inkscape:connector-curvature="0"
   3.495 +       id="path5221"
   3.496 +       d="m 666.85635,654.02605 c -65.78483,0 -65.78483,0 -65.78483,0"
   3.497 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
   3.498 +    <path
   3.499 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.500 +       d="m 668.02904,568.63925 c 0,13.68455 0,13.68455 0,13.68455"
   3.501 +       id="path5223"
   3.502 +       inkscape:connector-curvature="0"
   3.503 +       sodipodi:nodetypes="cc" />
   3.504 +    <path
   3.505 +       sodipodi:nodetypes="cc"
   3.506 +       inkscape:connector-curvature="0"
   3.507 +       id="path5229"
   3.508 +       d="m 602.02904,653.22687 c 0,5.5885 0,5.5885 0,5.5885"
   3.509 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.510 +    <path
   3.511 +       sodipodi:nodetypes="cc"
   3.512 +       inkscape:connector-curvature="0"
   3.513 +       id="path5235"
   3.514 +       d="m 668.02904,640.63925 c 0,13.68455 0,13.68455 0,13.68455"
   3.515 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.516 +    <text
   3.517 +       sodipodi:linespacing="100%"
   3.518 +       id="text5239"
   3.519 +       y="887.21881"
   3.520 +       x="75.689163"
   3.521 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.522 +       xml:space="preserve"
   3.523 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
   3.524 +         id="tspan5241"
   3.525 +         y="887.21881"
   3.526 +         x="75.689163"
   3.527 +         sodipodi:role="line"
   3.528 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan><tspan
   3.529 +         y="895.21881"
   3.530 +         x="75.689163"
   3.531 +         sodipodi:role="line"
   3.532 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.533 +         id="tspan5243">and switch</tspan><tspan
   3.534 +         y="903.21881"
   3.535 +         x="75.689163"
   3.536 +         sodipodi:role="line"
   3.537 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.538 +         id="tspan5245">to runtime</tspan></text>
   3.539 +    <path
   3.540 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.541 +       d="m 271.03421,642.02605 c 193.92439,0 193.92439,0 193.92439,0"
   3.542 +       id="path5247"
   3.543 +       inkscape:connector-curvature="0" />
   3.544 +    <text
   3.545 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
   3.546 +       xml:space="preserve"
   3.547 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.548 +       x="45.084938"
   3.549 +       y="918.83185"
   3.550 +       id="text5249"
   3.551 +       sodipodi:linespacing="100%"><tspan
   3.552 +         id="tspan5255"
   3.553 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.554 +         sodipodi:role="line"
   3.555 +         x="45.084938"
   3.556 +         y="918.83185">return</tspan><tspan
   3.557 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.558 +         sodipodi:role="line"
   3.559 +         x="45.084938"
   3.560 +         y="926.83185"
   3.561 +         id="tspan5261">from</tspan><tspan
   3.562 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.563 +         sodipodi:role="line"
   3.564 +         x="45.084938"
   3.565 +         y="934.83185"
   3.566 +         id="tspan5263">suspend</tspan></text>
   3.567 +    <text
   3.568 +       sodipodi:linespacing="100%"
   3.569 +       id="text5345"
   3.570 +       y="557.42468"
   3.571 +       x="667.38037"
   3.572 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.573 +       xml:space="preserve"><tspan
   3.574 +         id="tspan5349"
   3.575 +         y="557.42468"
   3.576 +         x="667.38037"
   3.577 +         sodipodi:role="line"
   3.578 +         style="text-align:center;text-anchor:middle;fill:#000080">PR primitive Fn</tspan><tspan
   3.579 +         y="567.42468"
   3.580 +         x="667.38037"
   3.581 +         sodipodi:role="line"
   3.582 +         style="text-align:center;text-anchor:middle;fill:#000080"
   3.583 +         id="tspan5353">to send request</tspan></text>
   3.584 +    <text
   3.585 +       sodipodi:linespacing="100%"
   3.586 +       id="text5355"
   3.587 +       y="554.71265"
   3.588 +       x="556.00226"
   3.589 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.590 +       xml:space="preserve"
   3.591 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
   3.592 +         id="tspan5357"
   3.593 +         y="554.71265"
   3.594 +         x="556.00226"
   3.595 +         sodipodi:role="line"
   3.596 +         style="font-size:8px;text-align:center;text-anchor:middle">normal call</tspan></text>
   3.597 +    <path
   3.598 +       sodipodi:nodetypes="cc"
   3.599 +       inkscape:connector-curvature="0"
   3.600 +       id="path5359"
   3.601 +       d="m 668.02904,689.28283 c 0,13.68455 0,13.68455 0,13.68455"
   3.602 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.603 +    <text
   3.604 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
   3.605 +       xml:space="preserve"
   3.606 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.607 +       x="-4.9486814"
   3.608 +       y="973.39014"
   3.609 +       id="text5363"
   3.610 +       sodipodi:linespacing="100%"><tspan
   3.611 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.612 +         sodipodi:role="line"
   3.613 +         x="-4.9486814"
   3.614 +         y="973.39014"
   3.615 +         id="tspan5365">suspend</tspan><tspan
   3.616 +         id="tspan5367"
   3.617 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.618 +         sodipodi:role="line"
   3.619 +         x="-4.9486814"
   3.620 +         y="981.39014">and switch</tspan><tspan
   3.621 +         id="tspan5369"
   3.622 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.623 +         sodipodi:role="line"
   3.624 +         x="-4.9486814"
   3.625 +         y="989.39014">to runtime</tspan></text>
   3.626 +    <text
   3.627 +       sodipodi:linespacing="100%"
   3.628 +       id="text5371"
   3.629 +       y="755.42468"
   3.630 +       x="604.59662"
   3.631 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.632 +       xml:space="preserve"><tspan
   3.633 +         id="tspan5373"
   3.634 +         y="755.42468"
   3.635 +         x="604.59662"
   3.636 +         sodipodi:role="line"
   3.637 +         style="text-align:center;text-anchor:middle;fill:#800000">Mutex Acquire</tspan><tspan
   3.638 +         id="tspan5375"
   3.639 +         y="765.42468"
   3.640 +         x="604.59662"
   3.641 +         sodipodi:role="line"
   3.642 +         style="text-align:center;text-anchor:middle;fill:#800000">wrapper Fn</tspan></text>
   3.643 +    <path
   3.644 +       inkscape:connector-curvature="0"
   3.645 +       id="path5379"
   3.646 +       d="m 514.24714,770.02605 c 85.56237,0 85.56237,0 85.56237,0"
   3.647 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
   3.648 +    <path
   3.649 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.650 +       d="m 602.77166,870.02605 c -88.41298,0 -88.41298,0 -88.41298,0"
   3.651 +       id="path5381"
   3.652 +       inkscape:connector-curvature="0" />
   3.653 +    <text
   3.654 +       xml:space="preserve"
   3.655 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.656 +       x="604.78461"
   3.657 +       y="882.71179"
   3.658 +       id="text5387"
   3.659 +       sodipodi:linespacing="100%"><tspan
   3.660 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.661 +         sodipodi:role="line"
   3.662 +         x="604.78461"
   3.663 +         y="882.71179"
   3.664 +         id="tspan5389">end VP</tspan><tspan
   3.665 +         id="tspan5391"
   3.666 +         style="text-align:center;text-anchor:middle;fill:#800000"
   3.667 +         sodipodi:role="line"
   3.668 +         x="604.78461"
   3.669 +         y="892.71179">wrapper Fn</tspan></text>
   3.670 +    <path
   3.671 +       inkscape:connector-curvature="0"
   3.672 +       id="path5393"
   3.673 +       d="m 602.289,902.02605 c 65.24023,0 65.24023,0 65.24023,0"
   3.674 +       style="fill:none;stroke:#000000;stroke-width:0.99999988;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
   3.675 +    <path
   3.676 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.677 +       d="m 602.02904,894.31985 c 0,7.35629 0,7.35629 0,7.35629"
   3.678 +       id="path5401"
   3.679 +       inkscape:connector-curvature="0"
   3.680 +       sodipodi:nodetypes="cc" />
   3.681 +    <path
   3.682 +       sodipodi:nodetypes="cc"
   3.683 +       inkscape:connector-curvature="0"
   3.684 +       id="path5403"
   3.685 +       d="m 602.02904,770.21843 c 0,19.4577 0,19.4577 0,19.4577"
   3.686 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.687 +    <path
   3.688 +       inkscape:connector-curvature="0"
   3.689 +       id="path5405"
   3.690 +       d="m 602.6148,790.02605 c 64.1779,0 64.1779,0 64.1779,0"
   3.691 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
   3.692 +    <path
   3.693 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.694 +       d="m 668.13452,866.02605 c -67.06305,0 -67.06305,0 -67.06305,0"
   3.695 +       id="path5407"
   3.696 +       inkscape:connector-curvature="0" />
   3.697 +    <path
   3.698 +       sodipodi:nodetypes="cc"
   3.699 +       inkscape:connector-curvature="0"
   3.700 +       id="path5409"
   3.701 +       d="m 668.02904,788.63925 c 0,13.68455 0,13.68455 0,13.68455"
   3.702 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.703 +    <path
   3.704 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.705 +       d="m 602.02904,865.22687 c 0,5.5885 0,5.5885 0,5.5885"
   3.706 +       id="path5411"
   3.707 +       inkscape:connector-curvature="0"
   3.708 +       sodipodi:nodetypes="cc" />
   3.709 +    <path
   3.710 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.711 +       d="m 668.02904,852.63925 c 0,13.68455 0,13.68455 0,13.68455"
   3.712 +       id="path5417"
   3.713 +       inkscape:connector-curvature="0"
   3.714 +       sodipodi:nodetypes="cc" />
   3.715 +    <text
   3.716 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
   3.717 +       xml:space="preserve"
   3.718 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.719 +       x="-74.503059"
   3.720 +       y="1045.2394"
   3.721 +       id="text5421"
   3.722 +       sodipodi:linespacing="100%"><tspan
   3.723 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.724 +         sodipodi:role="line"
   3.725 +         x="-74.503059"
   3.726 +         y="1045.2394"
   3.727 +         id="tspan5423">suspend</tspan><tspan
   3.728 +         id="tspan5425"
   3.729 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.730 +         sodipodi:role="line"
   3.731 +         x="-74.503059"
   3.732 +         y="1053.2394">and switch</tspan><tspan
   3.733 +         id="tspan5427"
   3.734 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.735 +         sodipodi:role="line"
   3.736 +         x="-74.503059"
   3.737 +         y="1061.2394">to runtime</tspan></text>
   3.738 +    <text
   3.739 +       sodipodi:linespacing="100%"
   3.740 +       id="text5431"
   3.741 +       y="1071.1506"
   3.742 +       x="-102.37076"
   3.743 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.744 +       xml:space="preserve"
   3.745 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
   3.746 +         y="1071.1506"
   3.747 +         x="-102.37076"
   3.748 +         sodipodi:role="line"
   3.749 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.750 +         id="tspan5433">return</tspan><tspan
   3.751 +         id="tspan5435"
   3.752 +         y="1079.1506"
   3.753 +         x="-102.37076"
   3.754 +         sodipodi:role="line"
   3.755 +         style="font-size:8px;text-align:center;text-anchor:middle">from</tspan><tspan
   3.756 +         id="tspan5437"
   3.757 +         y="1087.1506"
   3.758 +         x="-102.37076"
   3.759 +         sodipodi:role="line"
   3.760 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan></text>
   3.761 +    <text
   3.762 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
   3.763 +       xml:space="preserve"
   3.764 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.765 +       x="556.96783"
   3.766 +       y="774.71014"
   3.767 +       id="text5445"
   3.768 +       sodipodi:linespacing="100%"><tspan
   3.769 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.770 +         sodipodi:role="line"
   3.771 +         x="556.96783"
   3.772 +         y="774.71014"
   3.773 +         id="tspan5447">normal call</tspan></text>
   3.774 +    <path
   3.775 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.776 +       d="m 668.02904,901.28283 c 0,13.68455 0,13.68455 0,13.68455"
   3.777 +       id="path5449"
   3.778 +       inkscape:connector-curvature="0"
   3.779 +       sodipodi:nodetypes="cc" />
   3.780 +    <text
   3.781 +       sodipodi:linespacing="100%"
   3.782 +       id="text5453"
   3.783 +       y="1128.538"
   3.784 +       x="-152.3584"
   3.785 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.786 +       xml:space="preserve"
   3.787 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
   3.788 +         id="tspan5455"
   3.789 +         y="1128.538"
   3.790 +         x="-152.3584"
   3.791 +         sodipodi:role="line"
   3.792 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan><tspan
   3.793 +         y="1136.538"
   3.794 +         x="-152.3584"
   3.795 +         sodipodi:role="line"
   3.796 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.797 +         id="tspan5457">and switch</tspan><tspan
   3.798 +         y="1144.538"
   3.799 +         x="-152.3584"
   3.800 +         sodipodi:role="line"
   3.801 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.802 +         id="tspan5459">to runtime</tspan></text>
   3.803 +    <path
   3.804 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
   3.805 +       d="m 514.02904,870.31987 c 0,26.21842 0,26.21842 0,26.21842"
   3.806 +       id="path5461"
   3.807 +       inkscape:connector-curvature="0"
   3.808 +       sodipodi:nodetypes="cc" />
   3.809 +    <path
   3.810 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.811 +       d="m 514.2471,896.02605 c 87.58783,0 87.58783,0 87.58783,0"
   3.812 +       id="path5463"
   3.813 +       inkscape:connector-curvature="0" />
   3.814 +    <text
   3.815 +       sodipodi:linespacing="100%"
   3.816 +       id="text5465"
   3.817 +       y="688.71033"
   3.818 +       x="556.59009"
   3.819 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.820 +       xml:space="preserve"
   3.821 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
   3.822 +         id="tspan5467"
   3.823 +         y="688.71033"
   3.824 +         x="556.59009"
   3.825 +         sodipodi:role="line"
   3.826 +         style="font-size:8px;text-align:center;text-anchor:middle">normal call</tspan></text>
   3.827 +    <text
   3.828 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
   3.829 +       xml:space="preserve"
   3.830 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.831 +       x="557.52142"
   3.832 +       y="900.70972"
   3.833 +       id="text5469"
   3.834 +       sodipodi:linespacing="100%"><tspan
   3.835 +         style="font-size:8px;text-align:center;text-anchor:middle"
   3.836 +         sodipodi:role="line"
   3.837 +         x="557.52142"
   3.838 +         y="900.70972"
   3.839 +         id="tspan5471">normal call</tspan></text>
   3.840 +    <g
   3.841 +       transform="matrix(1,0,0,0.71130685,-16,282.52913)"
   3.842 +       id="g8395">
   3.843 +      <rect
   3.844 +         y="313.58255"
   3.845 +         x="310.40247"
   3.846 +         height="52.250008"
   3.847 +         width="65.276405"
   3.848 +         id="rect8397"
   3.849 +         style="fill:none;stroke:#000000;stroke-width:1.28076112;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
   3.850 +    </g>
   3.851 +    <path
   3.852 +       sodipodi:nodetypes="cc"
   3.853 +       inkscape:connector-curvature="0"
   3.854 +       id="path9273"
   3.855 +       d="m 514.02904,658.31987 c 0,26.21842 0,26.21842 0,26.21842"
   3.856 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
   3.857 +    <path
   3.858 +       sodipodi:nodetypes="cc"
   3.859 +       inkscape:connector-curvature="0"
   3.860 +       id="path9275"
   3.861 +       d="m 476.02904,544.79398 c 0,35.61575 0,35.61575 0,35.61575"
   3.862 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:none" />
   3.863 +    <path
   3.864 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
   3.865 +       d="m 468.24715,580.02605 c 13.34137,0 13.34137,0 13.34137,0"
   3.866 +       id="path9277"
   3.867 +       inkscape:connector-curvature="0" />
   3.868 +    <path
   3.869 +       inkscape:connector-curvature="0"
   3.870 +       id="path9279"
   3.871 +       d="m 481.57781,642.02605 c -13.14022,0 -13.14022,0 -13.14022,0"
   3.872 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
   3.873 +    <path
   3.874 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend)"
   3.875 +       d="m 476.02904,642.3113 c 0,57.40944 0,57.40944 0,57.40944"
   3.876 +       id="path9281"
   3.877 +       inkscape:connector-curvature="0"
   3.878 +       sodipodi:nodetypes="cc" />
   3.879 +    <path
   3.880 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
   3.881 +       d="m 481.57781,701.27417 c -13.14022,0 -13.14022,0 -13.14022,0"
   3.882 +       id="path9283"
   3.883 +       inkscape:connector-curvature="0" />
   3.884 +    <text
   3.885 +       transform="matrix(-0.01875541,0.9998241,-0.9998241,-0.01875541,0,0)"
   3.886 +       xml:space="preserve"
   3.887 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.888 +       x="615.90454"
   3.889 +       y="-469.42978"
   3.890 +       id="text9285"
   3.891 +       sodipodi:linespacing="100%"><tspan
   3.892 +         style="text-align:center;text-anchor:middle;fill:#008000"
   3.893 +         sodipodi:role="line"
   3.894 +         x="615.90454"
   3.895 +         y="-469.42978"
   3.896 +         id="tspan9287">Timeline of SeedVP</tspan></text>
   3.897 +    <text
   3.898 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
   3.899 +       xml:space="preserve"
   3.900 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.901 +       x="486.10849"
   3.902 +       y="579.01941"
   3.903 +       id="text9289"
   3.904 +       sodipodi:linespacing="100%"><tspan
   3.905 +         style="font-size:8px;text-align:start;text-anchor:start"
   3.906 +         sodipodi:role="line"
   3.907 +         x="486.10849"
   3.908 +         y="579.01941"
   3.909 +         id="tspan9291">suspend</tspan></text>
   3.910 +    <text
   3.911 +       sodipodi:linespacing="100%"
   3.912 +       id="text9293"
   3.913 +       y="643.01886"
   3.914 +       x="486.38965"
   3.915 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.916 +       xml:space="preserve"
   3.917 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
   3.918 +         id="tspan9295"
   3.919 +         y="643.01886"
   3.920 +         x="486.38965"
   3.921 +         sodipodi:role="line"
   3.922 +         style="font-size:8px;text-align:start;text-anchor:start">resume</tspan></text>
   3.923 +    <text
   3.924 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
   3.925 +       xml:space="preserve"
   3.926 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.927 +       x="486.64105"
   3.928 +       y="700.22437"
   3.929 +       id="text9297"
   3.930 +       sodipodi:linespacing="100%"><tspan
   3.931 +         style="font-size:8px;text-align:start;text-anchor:start"
   3.932 +         sodipodi:role="line"
   3.933 +         x="486.64105"
   3.934 +         y="700.22437"
   3.935 +         id="tspan9299">end</tspan></text>
   3.936 +    <text
   3.937 +       sodipodi:linespacing="100%"
   3.938 +       id="text9301"
   3.939 +       y="467.42465"
   3.940 +       x="63.167969"
   3.941 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.942 +       xml:space="preserve"><tspan
   3.943 +         id="tspan9303"
   3.944 +         y="467.42465"
   3.945 +         x="63.167969"
   3.946 +         sodipodi:role="line"
   3.947 +         style="text-align:center;text-anchor:middle;fill:#000080">Proto-Runtime</tspan><tspan
   3.948 +         id="tspan9305"
   3.949 +         y="477.42465"
   3.950 +         x="63.167969"
   3.951 +         sodipodi:role="line"
   3.952 +         style="text-align:center;text-anchor:middle;fill:#000080">Primitive</tspan><tspan
   3.953 +         y="487.42465"
   3.954 +         x="63.167969"
   3.955 +         sodipodi:role="line"
   3.956 +         style="text-align:center;text-anchor:middle;fill:#000080"
   3.957 +         id="tspan9307">Code</tspan></text>
   3.958 +    <text
   3.959 +       xml:space="preserve"
   3.960 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.961 +       x="65.380371"
   3.962 +       y="585.42468"
   3.963 +       id="text9309"
   3.964 +       sodipodi:linespacing="100%"><tspan
   3.965 +         id="tspan9313"
   3.966 +         style="text-align:center;text-anchor:middle;fill:#000080"
   3.967 +         sodipodi:role="line"
   3.968 +         x="65.380371"
   3.969 +         y="585.42468">PR primitive Fn</tspan><tspan
   3.970 +         style="text-align:center;text-anchor:middle;fill:#000080"
   3.971 +         sodipodi:role="line"
   3.972 +         x="65.380371"
   3.973 +         y="595.42468"
   3.974 +         id="tspan9317">to create VP</tspan></text>
   3.975 +    <path
   3.976 +       inkscape:connector-curvature="0"
   3.977 +       id="path9321"
   3.978 +       d="m 101.87628,587.85467 c 83.54552,12.98474 93.01032,9.74762 154.91834,33.79457 83.60123,32.47325 132.79902,111.74825 132.79902,111.74825"
   3.979 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.00000006, 3.00000006;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
   3.980 +       sodipodi:nodetypes="csc" />
   3.981 +    <g
   3.982 +       id="g9341"
   3.983 +       transform="translate(44.002789,336)">
   3.984 +      <rect
   3.985 +         y="389.58255"
   3.986 +         x="346.40247"
   3.987 +         height="37.165791"
   3.988 +         width="65.276405"
   3.989 +         id="rect9343"
   3.990 +         style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
   3.991 +      <text
   3.992 +         xml:space="preserve"
   3.993 +         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   3.994 +         x="379.35806"
   3.995 +         y="400.72159"
   3.996 +         id="text9345"
   3.997 +         sodipodi:linespacing="100%"><tspan
   3.998 +           style="text-align:center;text-anchor:middle;fill:#008000"
   3.999 +           sodipodi:role="line"
  3.1000 +           x="379.35806"
  3.1001 +           y="400.72159"
  3.1002 +           id="tspan9347">VP 1</tspan><tspan
  3.1003 +           style="font-size:8px;text-align:center;text-anchor:middle;fill:#008000"
  3.1004 +           sodipodi:role="line"
  3.1005 +           x="379.35806"
  3.1006 +           y="409.15637"
  3.1007 +           id="tspan9349">(created by</tspan><tspan
  3.1008 +           style="font-size:8px;text-align:center;text-anchor:middle;fill:#008000"
  3.1009 +           sodipodi:role="line"
  3.1010 +           x="379.35806"
  3.1011 +           y="417.15637"
  3.1012 +           id="tspan9355">application,</tspan><tspan
  3.1013 +           style="font-size:8px;text-align:center;text-anchor:middle;fill:#008000"
  3.1014 +           sodipodi:role="line"
  3.1015 +           x="379.35806"
  3.1016 +           y="425.15637"
  3.1017 +           id="tspan9746">on core 2)</tspan></text>
  3.1018 +    </g>
  3.1019 +    <g
  3.1020 +       transform="translate(44.002789,370)"
  3.1021 +       id="g9371" />
  3.1022 +    <text
  3.1023 +       xml:space="preserve"
  3.1024 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1025 +       x="202.41257"
  3.1026 +       y="541.42468"
  3.1027 +       id="text9512"
  3.1028 +       sodipodi:linespacing="100%"><tspan
  3.1029 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1030 +         sodipodi:role="line"
  3.1031 +         x="202.41257"
  3.1032 +         y="541.42468"
  3.1033 +         id="tspan9514">Master Fn</tspan></text>
  3.1034 +    <path
  3.1035 +       sodipodi:nodetypes="cc"
  3.1036 +       inkscape:connector-curvature="0"
  3.1037 +       id="path9522"
  3.1038 +       d="m 205.27805,580.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1039 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1040 +    <path
  3.1041 +       inkscape:connector-curvature="0"
  3.1042 +       id="path9524"
  3.1043 +       d="m 205.05995,586.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.1044 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1045 +    <path
  3.1046 +       sodipodi:nodetypes="cc"
  3.1047 +       inkscape:connector-curvature="0"
  3.1048 +       id="path9548"
  3.1049 +       d="m 137.27805,585.85824 c 0,11.43122 0,11.43122 0,11.43122"
  3.1050 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1051 +    <path
  3.1052 +       inkscape:connector-curvature="0"
  3.1053 +       id="path9550"
  3.1054 +       d="m 137.57123,598.02605 c -69.22157,0 -69.22157,0 -69.22157,0"
  3.1055 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1056 +    <path
  3.1057 +       sodipodi:nodetypes="cc"
  3.1058 +       inkscape:connector-curvature="0"
  3.1059 +       id="path9554"
  3.1060 +       d="m 205.27805,636.63925 c 0,4.54718 0,4.54718 0,4.54718"
  3.1061 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1062 +    <text
  3.1063 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
  3.1064 +       xml:space="preserve"
  3.1065 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1066 +       x="-300.35513"
  3.1067 +       y="609.46997"
  3.1068 +       id="text9566"
  3.1069 +       sodipodi:linespacing="100%"><tspan
  3.1070 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1071 +         sodipodi:role="line"
  3.1072 +         x="-300.35513"
  3.1073 +         y="609.46997"
  3.1074 +         id="tspan9568">suspend</tspan><tspan
  3.1075 +         id="tspan9570"
  3.1076 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1077 +         sodipodi:role="line"
  3.1078 +         x="-300.35513"
  3.1079 +         y="617.46997">and switch</tspan><tspan
  3.1080 +         id="tspan9572"
  3.1081 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1082 +         sodipodi:role="line"
  3.1083 +         x="-300.35513"
  3.1084 +         y="625.46997">to app VP</tspan></text>
  3.1085 +    <path
  3.1086 +       sodipodi:nodetypes="cc"
  3.1087 +       inkscape:connector-curvature="0"
  3.1088 +       id="path9614"
  3.1089 +       d="m 205.27805,580.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1090 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1091 +    <path
  3.1092 +       sodipodi:nodetypes="cc"
  3.1093 +       inkscape:connector-curvature="0"
  3.1094 +       id="path9616"
  3.1095 +       d="m 205.27805,580.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1096 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1097 +    <path
  3.1098 +       inkscape:connector-curvature="0"
  3.1099 +       id="path9648"
  3.1100 +       d="m 468.24715,545.42364 c 13.34137,0 13.34137,0 13.34137,0"
  3.1101 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.1102 +    <text
  3.1103 +       sodipodi:linespacing="100%"
  3.1104 +       id="text9654"
  3.1105 +       y="547.021"
  3.1106 +       x="485.96786"
  3.1107 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1108 +       xml:space="preserve"
  3.1109 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.1110 +         id="tspan9656"
  3.1111 +         y="547.021"
  3.1112 +         x="485.96786"
  3.1113 +         sodipodi:role="line"
  3.1114 +         style="font-size:8px;text-align:start;text-anchor:start">start</tspan></text>
  3.1115 +    <path
  3.1116 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:none"
  3.1117 +       d="m 476.02904,764.79398 c 0,35.61575 0,35.61575 0,35.61575"
  3.1118 +       id="path9658"
  3.1119 +       inkscape:connector-curvature="0"
  3.1120 +       sodipodi:nodetypes="cc" />
  3.1121 +    <path
  3.1122 +       inkscape:connector-curvature="0"
  3.1123 +       id="path9660"
  3.1124 +       d="m 468.24715,800.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.1125 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.1126 +    <path
  3.1127 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.1128 +       d="m 481.57781,854.02605 c -13.14022,0 -13.14022,0 -13.14022,0"
  3.1129 +       id="path9662"
  3.1130 +       inkscape:connector-curvature="0" />
  3.1131 +    <text
  3.1132 +       sodipodi:linespacing="100%"
  3.1133 +       id="text9668"
  3.1134 +       y="-467.51968"
  3.1135 +       x="833.97864"
  3.1136 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1137 +       xml:space="preserve"
  3.1138 +       transform="matrix(-0.01875541,0.9998241,-0.9998241,-0.01875541,0,0)"><tspan
  3.1139 +         y="-467.51968"
  3.1140 +         x="833.97864"
  3.1141 +         sodipodi:role="line"
  3.1142 +         style="text-align:center;text-anchor:middle;fill:#008000"
  3.1143 +         id="tspan9740">Timeline of VP 1</tspan></text>
  3.1144 +    <text
  3.1145 +       sodipodi:linespacing="100%"
  3.1146 +       id="text9672"
  3.1147 +       y="799.01776"
  3.1148 +       x="487.07419"
  3.1149 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1150 +       xml:space="preserve"
  3.1151 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.1152 +         id="tspan9674"
  3.1153 +         y="799.01776"
  3.1154 +         x="487.07419"
  3.1155 +         sodipodi:role="line"
  3.1156 +         style="font-size:8px;text-align:start;text-anchor:start">suspend</tspan></text>
  3.1157 +    <text
  3.1158 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.1159 +       xml:space="preserve"
  3.1160 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1161 +       x="487.32007"
  3.1162 +       y="855.01727"
  3.1163 +       id="text9676"
  3.1164 +       sodipodi:linespacing="100%"><tspan
  3.1165 +         style="font-size:8px;text-align:start;text-anchor:start"
  3.1166 +         sodipodi:role="line"
  3.1167 +         x="487.32007"
  3.1168 +         y="855.01727"
  3.1169 +         id="tspan9678">resume</tspan></text>
  3.1170 +    <path
  3.1171 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.1172 +       d="m 468.24715,765.42364 c 13.34137,0 13.34137,0 13.34137,0"
  3.1173 +       id="path9684"
  3.1174 +       inkscape:connector-curvature="0" />
  3.1175 +    <text
  3.1176 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.1177 +       xml:space="preserve"
  3.1178 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1179 +       x="486.93356"
  3.1180 +       y="767.01935"
  3.1181 +       id="text9686"
  3.1182 +       sodipodi:linespacing="100%"><tspan
  3.1183 +         style="font-size:8px;text-align:start;text-anchor:start"
  3.1184 +         sodipodi:role="line"
  3.1185 +         x="486.93356"
  3.1186 +         y="767.01935"
  3.1187 +         id="tspan9688">start</tspan></text>
  3.1188 +    <path
  3.1189 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:none"
  3.1190 +       d="m 266.02904,579.4311 c 0,61.88904 0,61.88904 0,61.88904"
  3.1191 +       id="path9690"
  3.1192 +       inkscape:connector-curvature="0"
  3.1193 +       sodipodi:nodetypes="cc" />
  3.1194 +    <path
  3.1195 +       inkscape:connector-curvature="0"
  3.1196 +       id="path9692"
  3.1197 +       d="m 258.24715,580.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.1198 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.1199 +    <path
  3.1200 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.1201 +       d="m 271.57781,642.02605 c -13.14022,0 -13.14022,0 -13.14022,0"
  3.1202 +       id="path9694"
  3.1203 +       inkscape:connector-curvature="0" />
  3.1204 +    <text
  3.1205 +       sodipodi:linespacing="100%"
  3.1206 +       id="text9700"
  3.1207 +       y="-276.31772"
  3.1208 +       x="625.34003"
  3.1209 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1210 +       xml:space="preserve"
  3.1211 +       transform="matrix(0.00239328,0.99999714,-0.99999714,0.00239328,0,0)"><tspan
  3.1212 +         y="-276.31772"
  3.1213 +         x="625.34003"
  3.1214 +         sodipodi:role="line"
  3.1215 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1216 +         id="tspan9736">Timeline of Master on core 1</tspan></text>
  3.1217 +    <text
  3.1218 +       sodipodi:linespacing="100%"
  3.1219 +       id="text9704"
  3.1220 +       y="642.12531"
  3.1221 +       x="262.08206"
  3.1222 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1223 +       xml:space="preserve"
  3.1224 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.1225 +         id="tspan9706"
  3.1226 +         y="642.12531"
  3.1227 +         x="262.08206"
  3.1228 +         sodipodi:role="line"
  3.1229 +         style="font-size:8px;text-align:end;text-anchor:end">suspend</tspan></text>
  3.1230 +    <text
  3.1231 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.1232 +       xml:space="preserve"
  3.1233 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1234 +       x="259.36166"
  3.1235 +       y="582.12573"
  3.1236 +       id="text9708"
  3.1237 +       sodipodi:linespacing="100%"><tspan
  3.1238 +         style="font-size:8px;text-align:end;text-anchor:end"
  3.1239 +         sodipodi:role="line"
  3.1240 +         x="259.36166"
  3.1241 +         y="582.12573"
  3.1242 +         id="tspan9710">resume</tspan></text>
  3.1243 +    <path
  3.1244 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.1245 +       d="m 258.24715,545.42364 c 13.34137,0 13.34137,0 13.34137,0"
  3.1246 +       id="path9716"
  3.1247 +       inkscape:connector-curvature="0" />
  3.1248 +    <text
  3.1249 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.1250 +       xml:space="preserve"
  3.1251 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1252 +       x="258.41916"
  3.1253 +       y="548.09222"
  3.1254 +       id="text9718"
  3.1255 +       sodipodi:linespacing="100%"><tspan
  3.1256 +         style="font-size:8px;text-align:end;text-anchor:end"
  3.1257 +         sodipodi:role="line"
  3.1258 +         x="258.41916"
  3.1259 +         y="548.09222"
  3.1260 +         id="tspan9720">start</tspan></text>
  3.1261 +    <path
  3.1262 +       sodipodi:nodetypes="cc"
  3.1263 +       inkscape:connector-curvature="0"
  3.1264 +       id="path9722"
  3.1265 +       d="m 266.02904,699.43112 c 0,10.33165 0,10.33165 0,10.33165"
  3.1266 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend)" />
  3.1267 +    <path
  3.1268 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.1269 +       d="m 258.24715,700.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.1270 +       id="path9724"
  3.1271 +       inkscape:connector-curvature="0" />
  3.1272 +    <text
  3.1273 +       sodipodi:linespacing="100%"
  3.1274 +       id="text9936"
  3.1275 +       y="786.9176"
  3.1276 +       x="644.57471"
  3.1277 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1278 +       xml:space="preserve"
  3.1279 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"><tspan
  3.1280 +         id="tspan9938"
  3.1281 +         y="786.9176"
  3.1282 +         x="644.57471"
  3.1283 +         sodipodi:role="line"
  3.1284 +         style="font-size:8px;text-align:center;text-anchor:middle">Call to dyn lib</tspan></text>
  3.1285 +    <text
  3.1286 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"
  3.1287 +       xml:space="preserve"
  3.1288 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1289 +       x="646.42737"
  3.1290 +       y="898.90509"
  3.1291 +       id="text9940"
  3.1292 +       sodipodi:linespacing="100%"><tspan
  3.1293 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1294 +         sodipodi:role="line"
  3.1295 +         x="646.42737"
  3.1296 +         y="898.90509"
  3.1297 +         id="tspan9942">Call to dyn lib</tspan></text>
  3.1298 +    <text
  3.1299 +       sodipodi:linespacing="100%"
  3.1300 +       id="text9944"
  3.1301 +       y="686.93219"
  3.1302 +       x="642.91998"
  3.1303 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1304 +       xml:space="preserve"
  3.1305 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"><tspan
  3.1306 +         id="tspan9946"
  3.1307 +         y="686.93219"
  3.1308 +         x="642.91998"
  3.1309 +         sodipodi:role="line"
  3.1310 +         style="font-size:8px;text-align:center;text-anchor:middle">Call to dyn lib</tspan></text>
  3.1311 +    <text
  3.1312 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"
  3.1313 +       xml:space="preserve"
  3.1314 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1315 +       x="115.47029"
  3.1316 +       y="603.64581"
  3.1317 +       id="text9948"
  3.1318 +       sodipodi:linespacing="100%"><tspan
  3.1319 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1320 +         sodipodi:role="line"
  3.1321 +         x="115.47029"
  3.1322 +         y="603.64581"
  3.1323 +         id="tspan9950">call to dyn lib</tspan></text>
  3.1324 +    <path
  3.1325 +       inkscape:connector-curvature="0"
  3.1326 +       id="path9952"
  3.1327 +       d="m 701.67453,642.02605 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.1328 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none" />
  3.1329 +    <path
  3.1330 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none"
  3.1331 +       d="m 701.67453,579.52981 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.1332 +       id="path9954"
  3.1333 +       inkscape:connector-curvature="0" />
  3.1334 +    <text
  3.1335 +       xml:space="preserve"
  3.1336 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1337 +       x="326.92542"
  3.1338 +       y="739.76575"
  3.1339 +       id="text9956"
  3.1340 +       sodipodi:linespacing="100%"><tspan
  3.1341 +         style="text-align:center;text-anchor:middle"
  3.1342 +         sodipodi:role="line"
  3.1343 +         id="tspan9958"
  3.1344 +         x="326.92542"
  3.1345 +         y="739.76575">Master</tspan><tspan
  3.1346 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1347 +         id="tspan9960"
  3.1348 +         sodipodi:role="line"
  3.1349 +         x="326.92542"
  3.1350 +         y="748.2005">(runtime system)</tspan><tspan
  3.1351 +         id="tspan9962"
  3.1352 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1353 +         sodipodi:role="line"
  3.1354 +         x="326.92542"
  3.1355 +         y="756.2005">on core 2</tspan></text>
  3.1356 +    <path
  3.1357 +       inkscape:connector-curvature="0"
  3.1358 +       id="path9970"
  3.1359 +       d="m 291.99816,758.02605 c -63.44071,0 -63.44071,0 -63.44071,0"
  3.1360 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.00000003, 3.00000003;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1361 +    <g
  3.1362 +       id="g9972"
  3.1363 +       transform="matrix(1,0,0,0.71130685,-16,504.52913)">
  3.1364 +      <rect
  3.1365 +         style="fill:none;stroke:#000000;stroke-width:1.28076112;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0"
  3.1366 +         id="rect9974"
  3.1367 +         width="65.276405"
  3.1368 +         height="52.250008"
  3.1369 +         x="310.40247"
  3.1370 +         y="313.58255" />
  3.1371 +    </g>
  3.1372 +    <text
  3.1373 +       sodipodi:linespacing="100%"
  3.1374 +       id="text9982"
  3.1375 +       y="761.42468"
  3.1376 +       x="202.41257"
  3.1377 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1378 +       xml:space="preserve"><tspan
  3.1379 +         id="tspan9984"
  3.1380 +         y="761.42468"
  3.1381 +         x="202.41257"
  3.1382 +         sodipodi:role="line"
  3.1383 +         style="text-align:center;text-anchor:middle;fill:#000080">Master Fn</tspan></text>
  3.1384 +    <path
  3.1385 +       sodipodi:nodetypes="cc"
  3.1386 +       inkscape:connector-curvature="0"
  3.1387 +       id="path10010"
  3.1388 +       d="m 266.02904,799.43113 c 0,54.93536 0,54.93536 0,54.93536"
  3.1389 +       style="fill:none;stroke:#000000;stroke-width:0.99999988px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:none" />
  3.1390 +    <path
  3.1391 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.1392 +       d="m 258.24715,800.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.1393 +       id="path10012"
  3.1394 +       inkscape:connector-curvature="0" />
  3.1395 +    <path
  3.1396 +       inkscape:connector-curvature="0"
  3.1397 +       id="path10014"
  3.1398 +       d="m 271.57781,854.02605 c -13.14022,0 -13.14022,0 -13.14022,0"
  3.1399 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.1400 +    <text
  3.1401 +       transform="matrix(0.00962718,0.99995366,-0.99995366,0.00962718,0,0)"
  3.1402 +       xml:space="preserve"
  3.1403 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1404 +       x="837.37762"
  3.1405 +       y="-273.75269"
  3.1406 +       id="text10018"
  3.1407 +       sodipodi:linespacing="100%"><tspan
  3.1408 +         id="tspan10020"
  3.1409 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1410 +         sodipodi:role="line"
  3.1411 +         x="837.37762"
  3.1412 +         y="-273.75269">Timeline of Master on core 2</tspan></text>
  3.1413 +    <path
  3.1414 +       inkscape:connector-curvature="0"
  3.1415 +       id="path10030"
  3.1416 +       d="m 258.24715,765.42364 c 13.34137,0 13.34137,0 13.34137,0"
  3.1417 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.1418 +    <path
  3.1419 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.00000002, 3.00000002;stroke-dashoffset:0"
  3.1420 +       d="m 205.27805,543.45805 c 0,3.23577 0,3.23577 0,3.23577"
  3.1421 +       id="path10054"
  3.1422 +       inkscape:connector-curvature="0"
  3.1423 +       sodipodi:nodetypes="cc" />
  3.1424 +    <path
  3.1425 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none"
  3.1426 +       d="m 701.67453,854.02605 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.1427 +       id="path10056"
  3.1428 +       inkscape:connector-curvature="0" />
  3.1429 +    <path
  3.1430 +       inkscape:connector-curvature="0"
  3.1431 +       id="path10058"
  3.1432 +       d="m 701.67453,799.52981 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.1433 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none" />
  3.1434 +    <text
  3.1435 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
  3.1436 +       xml:space="preserve"
  3.1437 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1438 +       x="-248.43199"
  3.1439 +       y="551.17401"
  3.1440 +       id="text10062"
  3.1441 +       sodipodi:linespacing="100%"><tspan
  3.1442 +         id="tspan10068"
  3.1443 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1444 +         sodipodi:role="line"
  3.1445 +         x="-248.43199"
  3.1446 +         y="551.17401">return from</tspan><tspan
  3.1447 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1448 +         sodipodi:role="line"
  3.1449 +         x="-248.43199"
  3.1450 +         y="559.17401"
  3.1451 +         id="tspan10074">suspend</tspan></text>
  3.1452 +    <path
  3.1453 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none"
  3.1454 +       d="m 701.67453,700.02605 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.1455 +       id="path10076"
  3.1456 +       inkscape:connector-curvature="0" />
  3.1457 +    <text
  3.1458 +       xml:space="preserve"
  3.1459 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1460 +       x="667.38037"
  3.1461 +       y="677.42468"
  3.1462 +       id="text10082"
  3.1463 +       sodipodi:linespacing="100%"><tspan
  3.1464 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1465 +         sodipodi:role="line"
  3.1466 +         x="667.38037"
  3.1467 +         y="677.42468"
  3.1468 +         id="tspan10084">PR primitive Fn</tspan><tspan
  3.1469 +         id="tspan10086"
  3.1470 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1471 +         sodipodi:role="line"
  3.1472 +         x="667.38037"
  3.1473 +         y="687.42468">to send request</tspan></text>
  3.1474 +    <text
  3.1475 +       sodipodi:linespacing="100%"
  3.1476 +       id="text10088"
  3.1477 +       y="777.42468"
  3.1478 +       x="667.38037"
  3.1479 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1480 +       xml:space="preserve"><tspan
  3.1481 +         id="tspan10090"
  3.1482 +         y="777.42468"
  3.1483 +         x="667.38037"
  3.1484 +         sodipodi:role="line"
  3.1485 +         style="text-align:center;text-anchor:middle;fill:#000080">PR primitive Fn</tspan><tspan
  3.1486 +         y="787.42468"
  3.1487 +         x="667.38037"
  3.1488 +         sodipodi:role="line"
  3.1489 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1490 +         id="tspan10092">to send request</tspan></text>
  3.1491 +    <text
  3.1492 +       xml:space="preserve"
  3.1493 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1494 +       x="667.38037"
  3.1495 +       y="889.42468"
  3.1496 +       id="text10094"
  3.1497 +       sodipodi:linespacing="100%"><tspan
  3.1498 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1499 +         sodipodi:role="line"
  3.1500 +         x="667.38037"
  3.1501 +         y="889.42468"
  3.1502 +         id="tspan10096">PR primitive Fn</tspan><tspan
  3.1503 +         id="tspan10098"
  3.1504 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1505 +         sodipodi:role="line"
  3.1506 +         x="667.38037"
  3.1507 +         y="899.42468">to send request</tspan></text>
  3.1508 +    <path
  3.1509 +       sodipodi:nodetypes="cc"
  3.1510 +       inkscape:connector-curvature="0"
  3.1511 +       id="path10100"
  3.1512 +       d="m 476.02904,854.3113 c 0,57.40944 0,57.40944 0,57.40944"
  3.1513 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend)" />
  3.1514 +    <path
  3.1515 +       inkscape:connector-curvature="0"
  3.1516 +       id="path10102"
  3.1517 +       d="m 481.57781,913.27417 c -13.14022,0 -13.14022,0 -13.14022,0"
  3.1518 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.1519 +    <text
  3.1520 +       sodipodi:linespacing="100%"
  3.1521 +       id="text10104"
  3.1522 +       y="912.22235"
  3.1523 +       x="487.57166"
  3.1524 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1525 +       xml:space="preserve"
  3.1526 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.1527 +         id="tspan10106"
  3.1528 +         y="912.22235"
  3.1529 +         x="487.57166"
  3.1530 +         sodipodi:role="line"
  3.1531 +         style="font-size:8px;text-align:start;text-anchor:start">end</tspan></text>
  3.1532 +    <path
  3.1533 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend)"
  3.1534 +       d="m 266.02904,911.43112 c 0,10.33165 0,10.33165 0,10.33165"
  3.1535 +       id="path10108"
  3.1536 +       inkscape:connector-curvature="0"
  3.1537 +       sodipodi:nodetypes="cc" />
  3.1538 +    <path
  3.1539 +       inkscape:connector-curvature="0"
  3.1540 +       id="path10110"
  3.1541 +       d="m 258.24715,912.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.1542 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.1543 +    <path
  3.1544 +       inkscape:connector-curvature="0"
  3.1545 +       id="path10116"
  3.1546 +       d="m 701.67453,912.02605 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.1547 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none" />
  3.1548 +    <text
  3.1549 +       sodipodi:linespacing="100%"
  3.1550 +       id="text10118"
  3.1551 +       y="702.12524"
  3.1552 +       x="259.88855"
  3.1553 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1554 +       xml:space="preserve"
  3.1555 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.1556 +         id="tspan10120"
  3.1557 +         y="702.12524"
  3.1558 +         x="259.88855"
  3.1559 +         sodipodi:role="line"
  3.1560 +         style="font-size:8px;text-align:end;text-anchor:end">resume</tspan></text>
  3.1561 +    <text
  3.1562 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.1563 +       xml:space="preserve"
  3.1564 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1565 +       x="260.32755"
  3.1566 +       y="802.12433"
  3.1567 +       id="text10122"
  3.1568 +       sodipodi:linespacing="100%"><tspan
  3.1569 +         style="font-size:8px;text-align:end;text-anchor:end"
  3.1570 +         sodipodi:role="line"
  3.1571 +         x="260.32755"
  3.1572 +         y="802.12433"
  3.1573 +         id="tspan10124">resume</tspan></text>
  3.1574 +    <text
  3.1575 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.1576 +       xml:space="preserve"
  3.1577 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1578 +       x="263.01257"
  3.1579 +       y="854.12366"
  3.1580 +       id="text10126"
  3.1581 +       sodipodi:linespacing="100%"><tspan
  3.1582 +         style="font-size:8px;text-align:end;text-anchor:end"
  3.1583 +         sodipodi:role="line"
  3.1584 +         x="263.01257"
  3.1585 +         y="854.12366"
  3.1586 +         id="tspan10128">suspend</tspan></text>
  3.1587 +    <text
  3.1588 +       sodipodi:linespacing="100%"
  3.1589 +       id="text10130"
  3.1590 +       y="768.09064"
  3.1591 +       x="259.38495"
  3.1592 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1593 +       xml:space="preserve"
  3.1594 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.1595 +         id="tspan10132"
  3.1596 +         y="768.09064"
  3.1597 +         x="259.38495"
  3.1598 +         sodipodi:role="line"
  3.1599 +         style="font-size:8px;text-align:end;text-anchor:end">start</tspan></text>
  3.1600 +    <text
  3.1601 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.1602 +       xml:space="preserve"
  3.1603 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1604 +       x="260.81906"
  3.1605 +       y="914.12299"
  3.1606 +       id="text10134"
  3.1607 +       sodipodi:linespacing="100%"><tspan
  3.1608 +         style="font-size:8px;text-align:end;text-anchor:end"
  3.1609 +         sodipodi:role="line"
  3.1610 +         x="260.81906"
  3.1611 +         y="914.12299"
  3.1612 +         id="tspan10136">resume</tspan></text>
  3.1613 +    <text
  3.1614 +       xml:space="preserve"
  3.1615 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1616 +       x="136.99219"
  3.1617 +       y="467.42465"
  3.1618 +       id="text10146"
  3.1619 +       sodipodi:linespacing="100%"><tspan
  3.1620 +         style="text-align:center;text-anchor:middle;fill:#800000"
  3.1621 +         sodipodi:role="line"
  3.1622 +         x="138.49609"
  3.1623 +         y="467.42465"
  3.1624 +         id="tspan10148">Language </tspan><tspan
  3.1625 +         id="tspan10150"
  3.1626 +         style="text-align:center;text-anchor:middle;fill:#800000"
  3.1627 +         sodipodi:role="line"
  3.1628 +         x="136.99219"
  3.1629 +         y="477.42465">Plugin</tspan><tspan
  3.1630 +         id="tspan10152"
  3.1631 +         style="text-align:center;text-anchor:middle;fill:#800000"
  3.1632 +         sodipodi:role="line"
  3.1633 +         x="136.99219"
  3.1634 +         y="487.42465">Code</tspan></text>
  3.1635 +    <text
  3.1636 +       xml:space="preserve"
  3.1637 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1638 +       x="203.16797"
  3.1639 +       y="467.42465"
  3.1640 +       id="text10154"
  3.1641 +       sodipodi:linespacing="100%"><tspan
  3.1642 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1643 +         sodipodi:role="line"
  3.1644 +         x="203.16797"
  3.1645 +         y="467.42465"
  3.1646 +         id="tspan10156">Proto-Runtime</tspan><tspan
  3.1647 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1648 +         sodipodi:role="line"
  3.1649 +         x="203.16797"
  3.1650 +         y="477.42465"
  3.1651 +         id="tspan10158">Master</tspan><tspan
  3.1652 +         id="tspan10160"
  3.1653 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.1654 +         sodipodi:role="line"
  3.1655 +         x="203.16797"
  3.1656 +         y="487.42465">Code</tspan></text>
  3.1657 +    <path
  3.1658 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1659 +       d="m 205.05995,586.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.1660 +       id="path10162"
  3.1661 +       inkscape:connector-curvature="0" />
  3.1662 +    <path
  3.1663 +       inkscape:connector-curvature="0"
  3.1664 +       id="path10164"
  3.1665 +       d="m 137.04459,636.02605 c 67.19099,0 67.19099,0 67.19099,0"
  3.1666 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1667 +    <path
  3.1668 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1669 +       d="m 67.525286,610.02605 c 69.221574,0 69.221574,0 69.221574,0"
  3.1670 +       id="path10166"
  3.1671 +       inkscape:connector-curvature="0" />
  3.1672 +    <path
  3.1673 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1674 +       d="m 67.27805,597.85824 c 0,13.94668 0,13.94668 0,13.94668"
  3.1675 +       id="path10168"
  3.1676 +       inkscape:connector-curvature="0"
  3.1677 +       sodipodi:nodetypes="cc" />
  3.1678 +    <path
  3.1679 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1680 +       d="m 137.04459,612.02605 c 67.19099,0 67.19099,0 67.19099,0"
  3.1681 +       id="path10172"
  3.1682 +       inkscape:connector-curvature="0" />
  3.1683 +    <path
  3.1684 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1685 +       d="m 205.27805,612.63926 c 0,11.23237 0,11.23237 0,11.23237"
  3.1686 +       id="path10174"
  3.1687 +       inkscape:connector-curvature="0"
  3.1688 +       sodipodi:nodetypes="cc" />
  3.1689 +    <path
  3.1690 +       inkscape:connector-curvature="0"
  3.1691 +       id="path10176"
  3.1692 +       d="m 205.05995,624.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.1693 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1694 +    <text
  3.1695 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"
  3.1696 +       xml:space="preserve"
  3.1697 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1698 +       x="191.85561"
  3.1699 +       y="626.38348"
  3.1700 +       id="text10178"
  3.1701 +       sodipodi:linespacing="100%"><tspan
  3.1702 +         style="font-size:7px;text-align:center;text-anchor:middle"
  3.1703 +         sodipodi:role="line"
  3.1704 +         x="191.85561"
  3.1705 +         y="626.38348"
  3.1706 +         id="tspan10180">call via Ptr to Dyn Lib Fn</tspan></text>
  3.1707 +    <path
  3.1708 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1709 +       d="m 137.27805,623.85824 c 0,11.43122 0,11.43122 0,11.43122"
  3.1710 +       id="path10182"
  3.1711 +       inkscape:connector-curvature="0"
  3.1712 +       sodipodi:nodetypes="cc" />
  3.1713 +    <text
  3.1714 +       sodipodi:linespacing="100%"
  3.1715 +       id="text10184"
  3.1716 +       y="588.38867"
  3.1717 +       x="191.22707"
  3.1718 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1719 +       xml:space="preserve"
  3.1720 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"><tspan
  3.1721 +         id="tspan10186"
  3.1722 +         y="588.38867"
  3.1723 +         x="191.22707"
  3.1724 +         sodipodi:role="line"
  3.1725 +         style="font-size:7px;text-align:center;text-anchor:middle">call via Ptr to Dyn Lib Fn</tspan></text>
  3.1726 +    <text
  3.1727 +       sodipodi:linespacing="100%"
  3.1728 +       id="text10188"
  3.1729 +       y="791.42468"
  3.1730 +       x="137.31689"
  3.1731 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1732 +       xml:space="preserve"><tspan
  3.1733 +         id="tspan10190"
  3.1734 +         y="791.42468"
  3.1735 +         x="137.31689"
  3.1736 +         sodipodi:role="line"
  3.1737 +         style="text-align:center;text-anchor:middle;fill:#800000">Lang Handler</tspan><tspan
  3.1738 +         id="tspan10192"
  3.1739 +         y="801.42468"
  3.1740 +         x="137.31689"
  3.1741 +         sodipodi:role="line"
  3.1742 +         style="text-align:center;text-anchor:middle;fill:#800000">for acq Mutex</tspan></text>
  3.1743 +    <path
  3.1744 +       sodipodi:nodetypes="cc"
  3.1745 +       inkscape:connector-curvature="0"
  3.1746 +       id="path10230"
  3.1747 +       d="m 205.27805,763.45805 c 0,3.23576 0,3.23576 0,3.23576"
  3.1748 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
  3.1749 +    <path
  3.1750 +       inkscape:connector-curvature="0"
  3.1751 +       id="path4879"
  3.1752 +       d="m 465.78297,700.02605 c -193.92439,0 -193.92439,0 -193.92439,0"
  3.1753 +       style="fill:none;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1754 +    <path
  3.1755 +       inkscape:connector-curvature="0"
  3.1756 +       id="path5395"
  3.1757 +       d="m 466.19516,580.02605 c -193.92439,0 -193.92439,0 -193.92439,0"
  3.1758 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1759 +    <path
  3.1760 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1761 +       d="m 466.19516,800.02605 c -193.92439,0 -193.92439,0 -193.92439,0"
  3.1762 +       id="path5397"
  3.1763 +       inkscape:connector-curvature="0" />
  3.1764 +    <path
  3.1765 +       inkscape:connector-curvature="0"
  3.1766 +       id="path5399"
  3.1767 +       d="m 466.19516,912.02605 c -193.92439,0 -193.92439,0 -193.92439,0"
  3.1768 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1769 +    <path
  3.1770 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1771 +       d="m 271.03421,854.02605 c 193.92439,0 193.92439,0 193.92439,0"
  3.1772 +       id="path5402"
  3.1773 +       inkscape:connector-curvature="0" />
  3.1774 +    <text
  3.1775 +       sodipodi:linespacing="100%"
  3.1776 +       id="text5410"
  3.1777 +       y="833.42468"
  3.1778 +       x="137.68066"
  3.1779 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1780 +       xml:space="preserve"><tspan
  3.1781 +         id="tspan5412"
  3.1782 +         y="833.42468"
  3.1783 +         x="137.68066"
  3.1784 +         sodipodi:role="line"
  3.1785 +         style="text-align:center;text-anchor:middle;fill:#800000">Assigner Fn</tspan></text>
  3.1786 +    <path
  3.1787 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1788 +       d="m 205.27805,800.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1789 +       id="path5420"
  3.1790 +       inkscape:connector-curvature="0"
  3.1791 +       sodipodi:nodetypes="cc" />
  3.1792 +    <path
  3.1793 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1794 +       d="m 205.05995,806.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.1795 +       id="path5422"
  3.1796 +       inkscape:connector-curvature="0" />
  3.1797 +    <path
  3.1798 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1799 +       d="m 137.27805,805.85824 c 0,11.43122 0,11.43122 0,11.43122"
  3.1800 +       id="path5424"
  3.1801 +       inkscape:connector-curvature="0"
  3.1802 +       sodipodi:nodetypes="cc" />
  3.1803 +    <path
  3.1804 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1805 +       d="m 205.27805,848.63925 c 0,4.54718 0,4.54718 0,4.54718"
  3.1806 +       id="path5428"
  3.1807 +       inkscape:connector-curvature="0"
  3.1808 +       sodipodi:nodetypes="cc" />
  3.1809 +    <text
  3.1810 +       sodipodi:linespacing="100%"
  3.1811 +       id="text5430"
  3.1812 +       y="761.78809"
  3.1813 +       x="-447.8111"
  3.1814 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1815 +       xml:space="preserve"
  3.1816 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.1817 +         id="tspan5432"
  3.1818 +         y="761.78809"
  3.1819 +         x="-447.8111"
  3.1820 +         sodipodi:role="line"
  3.1821 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan><tspan
  3.1822 +         y="769.78809"
  3.1823 +         x="-447.8111"
  3.1824 +         sodipodi:role="line"
  3.1825 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1826 +         id="tspan5434">and switch</tspan><tspan
  3.1827 +         y="777.78809"
  3.1828 +         x="-447.8111"
  3.1829 +         sodipodi:role="line"
  3.1830 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1831 +         id="tspan5436">to app VP</tspan></text>
  3.1832 +    <path
  3.1833 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1834 +       d="m 205.27805,800.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1835 +       id="path5438"
  3.1836 +       inkscape:connector-curvature="0"
  3.1837 +       sodipodi:nodetypes="cc" />
  3.1838 +    <path
  3.1839 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1840 +       d="m 205.27805,800.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1841 +       id="path5440"
  3.1842 +       inkscape:connector-curvature="0"
  3.1843 +       sodipodi:nodetypes="cc" />
  3.1844 +    <text
  3.1845 +       sodipodi:linespacing="100%"
  3.1846 +       id="text5446"
  3.1847 +       y="709.24017"
  3.1848 +       x="-401.45242"
  3.1849 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1850 +       xml:space="preserve"
  3.1851 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.1852 +         y="709.24017"
  3.1853 +         x="-401.45242"
  3.1854 +         sodipodi:role="line"
  3.1855 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1856 +         id="tspan5448">return from</tspan><tspan
  3.1857 +         id="tspan5450"
  3.1858 +         y="717.24017"
  3.1859 +         x="-401.45242"
  3.1860 +         sodipodi:role="line"
  3.1861 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan></text>
  3.1862 +    <path
  3.1863 +       inkscape:connector-curvature="0"
  3.1864 +       id="path5452"
  3.1865 +       d="m 205.05995,806.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.1866 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1867 +    <path
  3.1868 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1869 +       d="m 137.04459,848.02605 c 67.19099,0 67.19099,0 67.19099,0"
  3.1870 +       id="path5454"
  3.1871 +       inkscape:connector-curvature="0" />
  3.1872 +    <path
  3.1873 +       inkscape:connector-curvature="0"
  3.1874 +       id="path5460"
  3.1875 +       d="m 137.04459,820.02605 c 67.19099,0 67.19099,0 67.19099,0"
  3.1876 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.1877 +    <path
  3.1878 +       sodipodi:nodetypes="cc"
  3.1879 +       inkscape:connector-curvature="0"
  3.1880 +       id="path5462"
  3.1881 +       d="m 205.27805,820.63927 c 0,14.88618 0,14.88618 0,14.88618"
  3.1882 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1883 +    <path
  3.1884 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.1885 +       d="m 205.05995,836.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.1886 +       id="path5464"
  3.1887 +       inkscape:connector-curvature="0" />
  3.1888 +    <text
  3.1889 +       sodipodi:linespacing="100%"
  3.1890 +       id="text5466"
  3.1891 +       y="838.35522"
  3.1892 +       x="195.36256"
  3.1893 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1894 +       xml:space="preserve"
  3.1895 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"><tspan
  3.1896 +         id="tspan5468"
  3.1897 +         y="838.35522"
  3.1898 +         x="195.36256"
  3.1899 +         sodipodi:role="line"
  3.1900 +         style="font-size:7px;text-align:center;text-anchor:middle">call via Ptr to Dyn Lib Fn</tspan></text>
  3.1901 +    <path
  3.1902 +       sodipodi:nodetypes="cc"
  3.1903 +       inkscape:connector-curvature="0"
  3.1904 +       id="path5470"
  3.1905 +       d="m 137.27805,835.85824 c 0,11.43122 0,11.43122 0,11.43122"
  3.1906 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1907 +    <text
  3.1908 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"
  3.1909 +       xml:space="preserve"
  3.1910 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1911 +       x="194.86635"
  3.1912 +       y="808.35907"
  3.1913 +       id="text5472"
  3.1914 +       sodipodi:linespacing="100%"><tspan
  3.1915 +         style="font-size:7px;text-align:center;text-anchor:middle"
  3.1916 +         sodipodi:role="line"
  3.1917 +         x="194.86635"
  3.1918 +         y="808.35907"
  3.1919 +         id="tspan5474">call via Ptr to Dyn Lib Fn</tspan></text>
  3.1920 +    <text
  3.1921 +       sodipodi:linespacing="100%"
  3.1922 +       id="text5476"
  3.1923 +       y="713.48187"
  3.1924 +       x="-172.78647"
  3.1925 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1926 +       xml:space="preserve"
  3.1927 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.1928 +         y="713.48187"
  3.1929 +         x="-172.78647"
  3.1930 +         sodipodi:role="line"
  3.1931 +         style="font-size:18px;text-align:center;text-anchor:middle"
  3.1932 +         id="tspan5482">core 1</tspan></text>
  3.1933 +    <text
  3.1934 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
  3.1935 +       xml:space="preserve"
  3.1936 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1937 +       x="-321.0329"
  3.1938 +       y="865.03326"
  3.1939 +       id="text5488"
  3.1940 +       sodipodi:linespacing="100%"><tspan
  3.1941 +         id="tspan5490"
  3.1942 +         style="font-size:18px;text-align:center;text-anchor:middle"
  3.1943 +         sodipodi:role="line"
  3.1944 +         x="-321.0329"
  3.1945 +         y="865.03326">core 2</tspan></text>
  3.1946 +    <path
  3.1947 +       sodipodi:nodetypes="cc"
  3.1948 +       inkscape:connector-curvature="0"
  3.1949 +       id="path5492"
  3.1950 +       d="m 205.27805,914.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1951 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1952 +    <path
  3.1953 +       sodipodi:nodetypes="cc"
  3.1954 +       inkscape:connector-curvature="0"
  3.1955 +       id="path5494"
  3.1956 +       d="m 205.27805,914.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1957 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1958 +    <path
  3.1959 +       sodipodi:nodetypes="cc"
  3.1960 +       inkscape:connector-curvature="0"
  3.1961 +       id="path5496"
  3.1962 +       d="m 205.27805,914.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1963 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.1964 +    <text
  3.1965 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
  3.1966 +       xml:space="preserve"
  3.1967 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.1968 +       x="-480.74414"
  3.1969 +       y="791.14636"
  3.1970 +       id="text5498"
  3.1971 +       sodipodi:linespacing="100%"><tspan
  3.1972 +         id="tspan5500"
  3.1973 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1974 +         sodipodi:role="line"
  3.1975 +         x="-480.74414"
  3.1976 +         y="791.14636">return from</tspan><tspan
  3.1977 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.1978 +         sodipodi:role="line"
  3.1979 +         x="-480.74414"
  3.1980 +         y="799.14636"
  3.1981 +         id="tspan5502">suspend</tspan></text>
  3.1982 +    <path
  3.1983 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1984 +       d="m 205.27805,700.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1985 +       id="path5504"
  3.1986 +       inkscape:connector-curvature="0"
  3.1987 +       sodipodi:nodetypes="cc" />
  3.1988 +    <path
  3.1989 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1990 +       d="m 205.27805,700.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1991 +       id="path5506"
  3.1992 +       inkscape:connector-curvature="0"
  3.1993 +       sodipodi:nodetypes="cc" />
  3.1994 +    <path
  3.1995 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.1996 +       d="m 205.27805,700.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.1997 +       id="path5508"
  3.1998 +       inkscape:connector-curvature="0"
  3.1999 +       sodipodi:nodetypes="cc" />
  3.2000 +    <text
  3.2001 +       sodipodi:linespacing="100%"
  3.2002 +       id="text5510"
  3.2003 +       y="637.39264"
  3.2004 +       x="-331.89792"
  3.2005 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2006 +       xml:space="preserve"
  3.2007 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.2008 +         y="637.39264"
  3.2009 +         x="-331.89792"
  3.2010 +         sodipodi:role="line"
  3.2011 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2012 +         id="tspan5512">return from</tspan><tspan
  3.2013 +         id="tspan5514"
  3.2014 +         y="645.39264"
  3.2015 +         x="-331.89792"
  3.2016 +         sodipodi:role="line"
  3.2017 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan></text>
  3.2018 +    <path
  3.2019 +       inkscape:connector-curvature="0"
  3.2020 +       id="path5516"
  3.2021 +       d="m 271.03421,766.02605 c 193.92439,0 193.92439,0 193.92439,0"
  3.2022 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2023 +    <path
  3.2024 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2025 +       d="m 271.03421,546.02605 c 193.92439,0 193.92439,0 193.92439,0"
  3.2026 +       id="path5518"
  3.2027 +       inkscape:connector-curvature="0" />
  3.2028 +    <text
  3.2029 +       xml:space="preserve"
  3.2030 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2031 +       x="326.92542"
  3.2032 +       y="155.76575"
  3.2033 +       id="text3329"
  3.2034 +       sodipodi:linespacing="100%"><tspan
  3.2035 +         style="text-align:center;text-anchor:middle"
  3.2036 +         sodipodi:role="line"
  3.2037 +         id="tspan3331"
  3.2038 +         x="326.92542"
  3.2039 +         y="155.76575">PR OS thread</tspan><tspan
  3.2040 +         id="tspan3335"
  3.2041 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2042 +         sodipodi:role="line"
  3.2043 +         x="326.92542"
  3.2044 +         y="164.20053">(core1)</tspan></text>
  3.2045 +    <g
  3.2046 +       transform="translate(44.002789,-346)"
  3.2047 +       id="g3337">
  3.2048 +      <rect
  3.2049 +         y="389.58255"
  3.2050 +         x="346.40247"
  3.2051 +         height="37.165791"
  3.2052 +         width="65.276405"
  3.2053 +         id="rect3339"
  3.2054 +         style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
  3.2055 +      <text
  3.2056 +         xml:space="preserve"
  3.2057 +         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2058 +         x="379.00162"
  3.2059 +         y="400.15268"
  3.2060 +         id="text3341"
  3.2061 +         sodipodi:linespacing="100%"><tspan
  3.2062 +           id="tspan3349"
  3.2063 +           style="font-size:8px;text-align:center;text-anchor:middle;fill:#008000"
  3.2064 +           sodipodi:role="line"
  3.2065 +           x="379.00162"
  3.2066 +           y="400.15268"><tspan
  3.2067 +   id="tspan3641"
  3.2068 +   style="font-size:10px;text-align:center;text-anchor:middle;fill:#008000">Main </tspan>OS thread</tspan></text>
  3.2069 +    </g>
  3.2070 +    <text
  3.2071 +       xml:space="preserve"
  3.2072 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2073 +       x="517.16797"
  3.2074 +       y="81.424683"
  3.2075 +       id="text3351"
  3.2076 +       sodipodi:linespacing="100%"><tspan
  3.2077 +         style="text-align:center;text-anchor:middle;fill:#008000"
  3.2078 +         sodipodi:role="line"
  3.2079 +         x="517.16797"
  3.2080 +         y="81.424683"
  3.2081 +         id="tspan3353">main</tspan></text>
  3.2082 +    <text
  3.2083 +       sodipodi:linespacing="100%"
  3.2084 +       id="text3355"
  3.2085 +       y="273.42468"
  3.2086 +       x="137.31689"
  3.2087 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2088 +       xml:space="preserve"><tspan
  3.2089 +         id="tspan3357"
  3.2090 +         y="273.42468"
  3.2091 +         x="137.31689"
  3.2092 +         sodipodi:role="line"
  3.2093 +         style="text-align:center;text-anchor:middle;fill:#800000">Lang Handler</tspan><tspan
  3.2094 +         id="tspan3359"
  3.2095 +         y="283.42468"
  3.2096 +         x="137.31689"
  3.2097 +         sodipodi:role="line"
  3.2098 +         style="text-align:center;text-anchor:middle;fill:#800000">for create VP</tspan></text>
  3.2099 +    <text
  3.2100 +       sodipodi:linespacing="100%"
  3.2101 +       id="text3361"
  3.2102 +       y="321.42468"
  3.2103 +       x="137.68066"
  3.2104 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2105 +       xml:space="preserve"><tspan
  3.2106 +         id="tspan3363"
  3.2107 +         y="321.42468"
  3.2108 +         x="137.68066"
  3.2109 +         sodipodi:role="line"
  3.2110 +         style="text-align:center;text-anchor:middle;fill:#800000">Assigner Fn</tspan></text>
  3.2111 +    <path
  3.2112 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999996, 2.99999996;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2113 +       d="m 457.32649,78.02605 c 38.42351,0 38.42351,0 38.42351,0"
  3.2114 +       id="path3365"
  3.2115 +       inkscape:connector-curvature="0" />
  3.2116 +    <text
  3.2117 +       sodipodi:linespacing="100%"
  3.2118 +       id="text3367"
  3.2119 +       y="63.424683"
  3.2120 +       x="604.78461"
  3.2121 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2122 +       xml:space="preserve"><tspan
  3.2123 +         id="tspan3371"
  3.2124 +         y="63.424683"
  3.2125 +         x="604.78461"
  3.2126 +         sodipodi:role="line"
  3.2127 +         style="text-align:center;text-anchor:middle;fill:#800000">PR_start()</tspan></text>
  3.2128 +    <text
  3.2129 +       sodipodi:linespacing="100%"
  3.2130 +       id="text3373"
  3.2131 +       y="107.0096"
  3.2132 +       x="633.32379"
  3.2133 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2134 +       xml:space="preserve"
  3.2135 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"><tspan
  3.2136 +         id="tspan3375"
  3.2137 +         y="107.0096"
  3.2138 +         x="633.32379"
  3.2139 +         sodipodi:role="line"
  3.2140 +         style="font-size:8px;text-align:center;text-anchor:middle">Call to dyn lib</tspan></text>
  3.2141 +    <path
  3.2142 +       inkscape:connector-curvature="0"
  3.2143 +       id="path3377"
  3.2144 +       d="m 291.99816,238.02605 c -63.44071,0 -63.44071,0 -63.44071,0"
  3.2145 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.00000003, 3.00000003;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2146 +    <text
  3.2147 +       transform="matrix(0.97994281,-0.19927893,0.19927893,0.97994281,0,0)"
  3.2148 +       xml:space="preserve"
  3.2149 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2150 +       x="208.23512"
  3.2151 +       y="280.55707"
  3.2152 +       id="text3379"
  3.2153 +       sodipodi:linespacing="100%"><tspan
  3.2154 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2155 +         sodipodi:role="line"
  3.2156 +         x="208.23512"
  3.2157 +         y="280.55707"
  3.2158 +         id="tspan3381">Top Level Fn</tspan></text>
  3.2159 +    <text
  3.2160 +       sodipodi:linespacing="100%"
  3.2161 +       id="text3383"
  3.2162 +       y="166.44456"
  3.2163 +       x="460.02557"
  3.2164 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2165 +       xml:space="preserve"
  3.2166 +       transform="matrix(0.97994281,-0.19927893,0.19927893,0.97994281,0,0)"><tspan
  3.2167 +         id="tspan3385"
  3.2168 +         y="166.44456"
  3.2169 +         x="460.02557"
  3.2170 +         sodipodi:role="line"
  3.2171 +         style="font-size:8px;text-align:center;text-anchor:middle">birth Fn</tspan></text>
  3.2172 +    <path
  3.2173 +       sodipodi:nodetypes="cc"
  3.2174 +       inkscape:connector-curvature="0"
  3.2175 +       id="path3387"
  3.2176 +       d="m 514.02904,84.01894 c 0,5.6572 0,5.6572 0,5.6572"
  3.2177 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2178 +    <path
  3.2179 +       inkscape:connector-curvature="0"
  3.2180 +       id="path3389"
  3.2181 +       d="m 514.24714,90.02605 c 85.56237,0 85.56237,0 85.56237,0"
  3.2182 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2183 +    <path
  3.2184 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2185 +       d="m 602.77166,358.02605 c -88.41298,0 -88.41298,0 -88.41298,0"
  3.2186 +       id="path3391"
  3.2187 +       inkscape:connector-curvature="0" />
  3.2188 +    <path
  3.2189 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2190 +       d="m 514.02904,358.31987 c 0,26.21842 0,26.21842 0,26.21842"
  3.2191 +       id="path3393"
  3.2192 +       inkscape:connector-curvature="0"
  3.2193 +       sodipodi:nodetypes="cc" />
  3.2194 +    <path
  3.2195 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2196 +       d="m 514.2471,384.02605 c 87.58783,0 87.58783,0 87.58783,0"
  3.2197 +       id="path3395"
  3.2198 +       inkscape:connector-curvature="0" />
  3.2199 +    <text
  3.2200 +       xml:space="preserve"
  3.2201 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#800000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2202 +       x="604.78461"
  3.2203 +       y="370.71179"
  3.2204 +       id="text3397"
  3.2205 +       sodipodi:linespacing="100%"><tspan
  3.2206 +         style="text-align:center;text-anchor:middle;fill:#800000"
  3.2207 +         sodipodi:role="line"
  3.2208 +         x="604.78461"
  3.2209 +         y="370.71179"
  3.2210 +         id="tspan3399">end VP</tspan><tspan
  3.2211 +         id="tspan3401"
  3.2212 +         style="text-align:center;text-anchor:middle;fill:#800000"
  3.2213 +         sodipodi:role="line"
  3.2214 +         x="604.78461"
  3.2215 +         y="380.71179">wrapper Fn</tspan></text>
  3.2216 +    <path
  3.2217 +       inkscape:connector-curvature="0"
  3.2218 +       id="path3403"
  3.2219 +       d="m 602.26584,390.02605 c 63.99676,0 63.99676,0 63.99676,0"
  3.2220 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2221 +    <path
  3.2222 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2223 +       d="m 602.02904,382.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.2224 +       id="path3405"
  3.2225 +       inkscape:connector-curvature="0"
  3.2226 +       sodipodi:nodetypes="cc" />
  3.2227 +    <path
  3.2228 +       sodipodi:nodetypes="cc"
  3.2229 +       inkscape:connector-curvature="0"
  3.2230 +       id="path3407"
  3.2231 +       d="m 602.02904,90.21843 c 0,19.4577 0,19.4577 0,19.4577"
  3.2232 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2233 +    <path
  3.2234 +       inkscape:connector-curvature="0"
  3.2235 +       id="path3409"
  3.2236 +       d="m 602.24714,110.02605 c 64.71029,0 64.71029,0 64.71029,0"
  3.2237 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2238 +    <path
  3.2239 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2240 +       d="m 666.85635,354.02605 c -65.78483,0 -65.78483,0 -65.78483,0"
  3.2241 +       id="path3411"
  3.2242 +       inkscape:connector-curvature="0" />
  3.2243 +    <path
  3.2244 +       sodipodi:nodetypes="cc"
  3.2245 +       inkscape:connector-curvature="0"
  3.2246 +       id="path3413"
  3.2247 +       d="m 668.02904,108.63925 c 0,13.68455 0,13.68455 0,13.68455"
  3.2248 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2249 +    <path
  3.2250 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2251 +       d="m 602.02904,353.22687 c 0,5.5885 0,5.5885 0,5.5885"
  3.2252 +       id="path3415"
  3.2253 +       inkscape:connector-curvature="0"
  3.2254 +       sodipodi:nodetypes="cc" />
  3.2255 +    <path
  3.2256 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2257 +       d="m 668.02904,340.63925 c 0,13.68455 0,13.68455 0,13.68455"
  3.2258 +       id="path3417"
  3.2259 +       inkscape:connector-curvature="0"
  3.2260 +       sodipodi:nodetypes="cc" />
  3.2261 +    <text
  3.2262 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
  3.2263 +       xml:space="preserve"
  3.2264 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2265 +       x="395.64081"
  3.2266 +       y="556.71735"
  3.2267 +       id="text3419"
  3.2268 +       sodipodi:linespacing="100%"><tspan
  3.2269 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2270 +         sodipodi:role="line"
  3.2271 +         x="395.64081"
  3.2272 +         y="556.71735"
  3.2273 +         id="tspan3421">suspend</tspan><tspan
  3.2274 +         id="tspan3423"
  3.2275 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2276 +         sodipodi:role="line"
  3.2277 +         x="395.64081"
  3.2278 +         y="564.71735">and switch</tspan><tspan
  3.2279 +         id="tspan3425"
  3.2280 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2281 +         sodipodi:role="line"
  3.2282 +         x="395.64081"
  3.2283 +         y="572.71735">to runtime</tspan></text>
  3.2284 +    <path
  3.2285 +       inkscape:connector-curvature="0"
  3.2286 +       id="path3427"
  3.2287 +       d="m 271.03421,342.02605 c 193.92439,0 193.92439,0 193.92439,0"
  3.2288 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2289 +    <text
  3.2290 +       sodipodi:linespacing="100%"
  3.2291 +       id="text3429"
  3.2292 +       y="703.28741"
  3.2293 +       x="253.74896"
  3.2294 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2295 +       xml:space="preserve"
  3.2296 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.2297 +         y="703.28741"
  3.2298 +         x="253.74896"
  3.2299 +         sodipodi:role="line"
  3.2300 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2301 +         id="tspan3431">return</tspan><tspan
  3.2302 +         id="tspan3433"
  3.2303 +         y="711.28741"
  3.2304 +         x="253.74896"
  3.2305 +         sodipodi:role="line"
  3.2306 +         style="font-size:8px;text-align:center;text-anchor:middle">from</tspan><tspan
  3.2307 +         id="tspan3435"
  3.2308 +         y="719.28741"
  3.2309 +         x="253.74896"
  3.2310 +         sodipodi:role="line"
  3.2311 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan></text>
  3.2312 +    <text
  3.2313 +       xml:space="preserve"
  3.2314 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2315 +       x="601.38037"
  3.2316 +       y="83.424683"
  3.2317 +       id="text3437"
  3.2318 +       sodipodi:linespacing="100%"><tspan
  3.2319 +         id="tspan3441"
  3.2320 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.2321 +         sodipodi:role="line"
  3.2322 +         x="601.38037"
  3.2323 +         y="83.424683">PR__start()</tspan></text>
  3.2324 +    <text
  3.2325 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.2326 +       xml:space="preserve"
  3.2327 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2328 +       x="553.98218"
  3.2329 +       y="94.71698"
  3.2330 +       id="text3443"
  3.2331 +       sodipodi:linespacing="100%"><tspan
  3.2332 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2333 +         sodipodi:role="line"
  3.2334 +         x="553.98218"
  3.2335 +         y="94.71698"
  3.2336 +         id="tspan3445">normal call</tspan></text>
  3.2337 +    <path
  3.2338 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2339 +       d="m 668.02904,389.28283 c 0,13.68455 0,13.68455 0,13.68455"
  3.2340 +       id="path3447"
  3.2341 +       inkscape:connector-curvature="0"
  3.2342 +       sodipodi:nodetypes="cc" />
  3.2343 +    <text
  3.2344 +       sodipodi:linespacing="100%"
  3.2345 +       id="text3449"
  3.2346 +       y="757.8457"
  3.2347 +       x="203.71536"
  3.2348 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2349 +       xml:space="preserve"
  3.2350 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.2351 +         id="tspan3451"
  3.2352 +         y="757.8457"
  3.2353 +         x="203.71536"
  3.2354 +         sodipodi:role="line"
  3.2355 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan><tspan
  3.2356 +         y="765.8457"
  3.2357 +         x="203.71536"
  3.2358 +         sodipodi:role="line"
  3.2359 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2360 +         id="tspan3453">and switch</tspan><tspan
  3.2361 +         y="773.8457"
  3.2362 +         x="203.71536"
  3.2363 +         sodipodi:role="line"
  3.2364 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2365 +         id="tspan3455">to runtime</tspan></text>
  3.2366 +    <text
  3.2367 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.2368 +       xml:space="preserve"
  3.2369 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2370 +       x="555.27264"
  3.2371 +       y="388.71307"
  3.2372 +       id="text3457"
  3.2373 +       sodipodi:linespacing="100%"><tspan
  3.2374 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2375 +         sodipodi:role="line"
  3.2376 +         x="555.27264"
  3.2377 +         y="388.71307"
  3.2378 +         id="tspan3459">normal call</tspan></text>
  3.2379 +    <g
  3.2380 +       id="g3461"
  3.2381 +       transform="matrix(1,0,0,0.71130685,-16,-77.47087)">
  3.2382 +      <rect
  3.2383 +         style="fill:none;stroke:#000000;stroke-width:1.28076112;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0"
  3.2384 +         id="rect3463"
  3.2385 +         width="65.276405"
  3.2386 +         height="52.250008"
  3.2387 +         x="310.40247"
  3.2388 +         y="313.58255" />
  3.2389 +    </g>
  3.2390 +    <path
  3.2391 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2392 +       d="m 514.02904,358.31987 c 0,26.21842 0,26.21842 0,26.21842"
  3.2393 +       id="path3465"
  3.2394 +       inkscape:connector-curvature="0"
  3.2395 +       sodipodi:nodetypes="cc" />
  3.2396 +    <path
  3.2397 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:none"
  3.2398 +       d="m 476.02904,84.79398 c 0,35.61575 0,35.61575 0,35.61575"
  3.2399 +       id="path3467"
  3.2400 +       inkscape:connector-curvature="0"
  3.2401 +       sodipodi:nodetypes="cc" />
  3.2402 +    <path
  3.2403 +       inkscape:connector-curvature="0"
  3.2404 +       id="path3469"
  3.2405 +       d="m 468.24715,120.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.2406 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.2407 +    <path
  3.2408 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.2409 +       d="m 481.57781,342.02605 c -13.14022,0 -13.14022,0 -13.14022,0"
  3.2410 +       id="path3471"
  3.2411 +       inkscape:connector-curvature="0" />
  3.2412 +    <path
  3.2413 +       sodipodi:nodetypes="cc"
  3.2414 +       inkscape:connector-curvature="0"
  3.2415 +       id="path3473"
  3.2416 +       d="m 476.02904,342.3113 c 0,57.40944 0,57.40944 0,57.40944"
  3.2417 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend)" />
  3.2418 +    <path
  3.2419 +       inkscape:connector-curvature="0"
  3.2420 +       id="path3475"
  3.2421 +       d="m 481.57781,401.27417 c -13.14022,0 -13.14022,0 -13.14022,0"
  3.2422 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.2423 +    <text
  3.2424 +       sodipodi:linespacing="100%"
  3.2425 +       id="text3477"
  3.2426 +       y="-463.80295"
  3.2427 +       x="315.95734"
  3.2428 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#008000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2429 +       xml:space="preserve"
  3.2430 +       transform="matrix(-0.01875541,0.9998241,-0.9998241,-0.01875541,0,0)"><tspan
  3.2431 +         id="tspan3479"
  3.2432 +         y="-463.80295"
  3.2433 +         x="315.95734"
  3.2434 +         sodipodi:role="line"
  3.2435 +         style="text-align:center;text-anchor:middle;fill:#008000">Timeline of SeedVP</tspan></text>
  3.2436 +    <text
  3.2437 +       sodipodi:linespacing="100%"
  3.2438 +       id="text3481"
  3.2439 +       y="119.02372"
  3.2440 +       x="484.08911"
  3.2441 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2442 +       xml:space="preserve"
  3.2443 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.2444 +         id="tspan3483"
  3.2445 +         y="119.02372"
  3.2446 +         x="484.08911"
  3.2447 +         sodipodi:role="line"
  3.2448 +         style="font-size:8px;text-align:start;text-anchor:start">suspend</tspan></text>
  3.2449 +    <text
  3.2450 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.2451 +       xml:space="preserve"
  3.2452 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2453 +       x="485.07266"
  3.2454 +       y="343.02161"
  3.2455 +       id="text3485"
  3.2456 +       sodipodi:linespacing="100%"><tspan
  3.2457 +         style="font-size:8px;text-align:start;text-anchor:start"
  3.2458 +         sodipodi:role="line"
  3.2459 +         x="485.07266"
  3.2460 +         y="343.02161"
  3.2461 +         id="tspan3487">resume</tspan></text>
  3.2462 +    <text
  3.2463 +       sodipodi:linespacing="100%"
  3.2464 +       id="text3489"
  3.2465 +       y="400.22711"
  3.2466 +       x="485.32407"
  3.2467 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2468 +       xml:space="preserve"
  3.2469 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.2470 +         id="tspan3491"
  3.2471 +         y="400.22711"
  3.2472 +         x="485.32407"
  3.2473 +         sodipodi:role="line"
  3.2474 +         style="font-size:8px;text-align:start;text-anchor:start">end</tspan></text>
  3.2475 +    <text
  3.2476 +       sodipodi:linespacing="100%"
  3.2477 +       id="text3493"
  3.2478 +       y="285.42468"
  3.2479 +       x="65.380371"
  3.2480 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2481 +       xml:space="preserve"><tspan
  3.2482 +         y="285.42468"
  3.2483 +         x="65.380371"
  3.2484 +         sodipodi:role="line"
  3.2485 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.2486 +         id="tspan3495">PR primitive Fn</tspan><tspan
  3.2487 +         id="tspan3497"
  3.2488 +         y="295.42468"
  3.2489 +         x="65.380371"
  3.2490 +         sodipodi:role="line"
  3.2491 +         style="text-align:center;text-anchor:middle;fill:#000080">to create VP</tspan></text>
  3.2492 +    <text
  3.2493 +       sodipodi:linespacing="100%"
  3.2494 +       id="text3499"
  3.2495 +       y="241.42468"
  3.2496 +       x="202.41257"
  3.2497 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2498 +       xml:space="preserve"><tspan
  3.2499 +         id="tspan3501"
  3.2500 +         y="241.42468"
  3.2501 +         x="202.41257"
  3.2502 +         sodipodi:role="line"
  3.2503 +         style="text-align:center;text-anchor:middle;fill:#000080">Master Fn</tspan></text>
  3.2504 +    <path
  3.2505 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2506 +       d="m 205.27805,280.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.2507 +       id="path3503"
  3.2508 +       inkscape:connector-curvature="0"
  3.2509 +       sodipodi:nodetypes="cc" />
  3.2510 +    <path
  3.2511 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2512 +       d="m 205.05995,286.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.2513 +       id="path3505"
  3.2514 +       inkscape:connector-curvature="0" />
  3.2515 +    <path
  3.2516 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2517 +       d="m 137.27805,285.85824 c 0,11.43122 0,11.43122 0,11.43122"
  3.2518 +       id="path3507"
  3.2519 +       inkscape:connector-curvature="0"
  3.2520 +       sodipodi:nodetypes="cc" />
  3.2521 +    <path
  3.2522 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2523 +       d="m 137.57123,298.02605 c -69.22157,0 -69.22157,0 -69.22157,0"
  3.2524 +       id="path3509"
  3.2525 +       inkscape:connector-curvature="0" />
  3.2526 +    <path
  3.2527 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2528 +       d="m 205.27805,336.63925 c 0,4.54718 0,4.54718 0,4.54718"
  3.2529 +       id="path3511"
  3.2530 +       inkscape:connector-curvature="0"
  3.2531 +       sodipodi:nodetypes="cc" />
  3.2532 +    <text
  3.2533 +       sodipodi:linespacing="100%"
  3.2534 +       id="text3513"
  3.2535 +       y="393.92554"
  3.2536 +       x="-91.691048"
  3.2537 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2538 +       xml:space="preserve"
  3.2539 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.2540 +         id="tspan3515"
  3.2541 +         y="393.92554"
  3.2542 +         x="-91.691048"
  3.2543 +         sodipodi:role="line"
  3.2544 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan><tspan
  3.2545 +         y="401.92554"
  3.2546 +         x="-91.691048"
  3.2547 +         sodipodi:role="line"
  3.2548 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2549 +         id="tspan3517">and switch</tspan><tspan
  3.2550 +         y="409.92554"
  3.2551 +         x="-91.691048"
  3.2552 +         sodipodi:role="line"
  3.2553 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2554 +         id="tspan3519">to app VP</tspan></text>
  3.2555 +    <path
  3.2556 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2557 +       d="m 205.27805,280.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.2558 +       id="path3521"
  3.2559 +       inkscape:connector-curvature="0"
  3.2560 +       sodipodi:nodetypes="cc" />
  3.2561 +    <path
  3.2562 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
  3.2563 +       d="m 205.27805,280.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.2564 +       id="path3523"
  3.2565 +       inkscape:connector-curvature="0"
  3.2566 +       sodipodi:nodetypes="cc" />
  3.2567 +    <path
  3.2568 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.2569 +       d="m 468.24715,85.42364 c 13.34137,0 13.34137,0 13.34137,0"
  3.2570 +       id="path3525"
  3.2571 +       inkscape:connector-curvature="0" />
  3.2572 +    <text
  3.2573 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.2574 +       xml:space="preserve"
  3.2575 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2576 +       x="483.94849"
  3.2577 +       y="87.025322"
  3.2578 +       id="text3527"
  3.2579 +       sodipodi:linespacing="100%"><tspan
  3.2580 +         style="font-size:8px;text-align:start;text-anchor:start"
  3.2581 +         sodipodi:role="line"
  3.2582 +         x="483.94849"
  3.2583 +         y="87.025322"
  3.2584 +         id="tspan3529">start</tspan></text>
  3.2585 +    <path
  3.2586 +       sodipodi:nodetypes="cc"
  3.2587 +       inkscape:connector-curvature="0"
  3.2588 +       id="path3531"
  3.2589 +       d="m 266.02904,279.4311 c 0,61.88904 0,61.88904 0,61.88904"
  3.2590 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:none" />
  3.2591 +    <path
  3.2592 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none"
  3.2593 +       d="m 258.24715,280.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.2594 +       id="path3533"
  3.2595 +       inkscape:connector-curvature="0" />
  3.2596 +    <path
  3.2597 +       inkscape:connector-curvature="0"
  3.2598 +       id="path3535"
  3.2599 +       d="m 271.57781,342.02605 c -13.14022,0 -13.14022,0 -13.14022,0"
  3.2600 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.2601 +    <text
  3.2602 +       transform="matrix(0.00239328,0.99999714,-0.99999714,0.00239328,0,0)"
  3.2603 +       xml:space="preserve"
  3.2604 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2605 +       x="325.34094"
  3.2606 +       y="-277.03549"
  3.2607 +       id="text3537"
  3.2608 +       sodipodi:linespacing="100%"><tspan
  3.2609 +         id="tspan3539"
  3.2610 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.2611 +         sodipodi:role="line"
  3.2612 +         x="325.34094"
  3.2613 +         y="-277.03549">Timeline of Master on core 1</tspan></text>
  3.2614 +    <text
  3.2615 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.2616 +       xml:space="preserve"
  3.2617 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2618 +       x="260.76508"
  3.2619 +       y="342.12805"
  3.2620 +       id="text3541"
  3.2621 +       sodipodi:linespacing="100%"><tspan
  3.2622 +         style="font-size:8px;text-align:end;text-anchor:end"
  3.2623 +         sodipodi:role="line"
  3.2624 +         x="260.76508"
  3.2625 +         y="342.12805"
  3.2626 +         id="tspan3543">suspend</tspan></text>
  3.2627 +    <text
  3.2628 +       sodipodi:linespacing="100%"
  3.2629 +       id="text3545"
  3.2630 +       y="282.12848"
  3.2631 +       x="258.04468"
  3.2632 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2633 +       xml:space="preserve"
  3.2634 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.2635 +         id="tspan3547"
  3.2636 +         y="282.12848"
  3.2637 +         x="258.04468"
  3.2638 +         sodipodi:role="line"
  3.2639 +         style="font-size:8px;text-align:end;text-anchor:end">resume</tspan></text>
  3.2640 +    <path
  3.2641 +       inkscape:connector-curvature="0"
  3.2642 +       id="path3549"
  3.2643 +       d="m 258.24715,245.42364 c 13.34137,0 13.34137,0 13.34137,0"
  3.2644 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.2645 +    <text
  3.2646 +       sodipodi:linespacing="100%"
  3.2647 +       id="text3551"
  3.2648 +       y="248.09499"
  3.2649 +       x="257.10217"
  3.2650 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2651 +       xml:space="preserve"
  3.2652 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"><tspan
  3.2653 +         id="tspan3553"
  3.2654 +         y="248.09499"
  3.2655 +         x="257.10217"
  3.2656 +         sodipodi:role="line"
  3.2657 +         style="font-size:8px;text-align:end;text-anchor:end">start</tspan></text>
  3.2658 +    <path
  3.2659 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend)"
  3.2660 +       d="m 266.02904,399.43112 c 0,10.33165 0,10.33165 0,10.33165"
  3.2661 +       id="path3555"
  3.2662 +       inkscape:connector-curvature="0"
  3.2663 +       sodipodi:nodetypes="cc" />
  3.2664 +    <path
  3.2665 +       inkscape:connector-curvature="0"
  3.2666 +       id="path3557"
  3.2667 +       d="m 258.24715,400.02605 c 13.34137,0 13.34137,0 13.34137,0"
  3.2668 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:none" />
  3.2669 +    <text
  3.2670 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"
  3.2671 +       xml:space="preserve"
  3.2672 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2673 +       x="637.95691"
  3.2674 +       y="386.97339"
  3.2675 +       id="text3559"
  3.2676 +       sodipodi:linespacing="100%"><tspan
  3.2677 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2678 +         sodipodi:role="line"
  3.2679 +         x="637.95691"
  3.2680 +         y="386.97339"
  3.2681 +         id="tspan3561">Call to dyn lib</tspan></text>
  3.2682 +    <text
  3.2683 +       sodipodi:linespacing="100%"
  3.2684 +       id="text3563"
  3.2685 +       y="303.68701"
  3.2686 +       x="110.50755"
  3.2687 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2688 +       xml:space="preserve"
  3.2689 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"><tspan
  3.2690 +         id="tspan3565"
  3.2691 +         y="303.68701"
  3.2692 +         x="110.50755"
  3.2693 +         sodipodi:role="line"
  3.2694 +         style="font-size:8px;text-align:center;text-anchor:middle">call to dyn lib</tspan></text>
  3.2695 +    <path
  3.2696 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none"
  3.2697 +       d="m 701.67453,342.02605 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.2698 +       id="path3567"
  3.2699 +       inkscape:connector-curvature="0" />
  3.2700 +    <path
  3.2701 +       inkscape:connector-curvature="0"
  3.2702 +       id="path3569"
  3.2703 +       d="m 701.67453,279.52981 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.2704 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none" />
  3.2705 +    <path
  3.2706 +       sodipodi:nodetypes="cc"
  3.2707 +       inkscape:connector-curvature="0"
  3.2708 +       id="path3571"
  3.2709 +       d="m 205.27805,243.45805 c 0,3.23577 0,3.23577 0,3.23577"
  3.2710 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.00000002, 3.00000002;stroke-dashoffset:0" />
  3.2711 +    <text
  3.2712 +       sodipodi:linespacing="100%"
  3.2713 +       id="text3573"
  3.2714 +       y="335.62958"
  3.2715 +       x="-39.767914"
  3.2716 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2717 +       xml:space="preserve"
  3.2718 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"><tspan
  3.2719 +         y="335.62958"
  3.2720 +         x="-39.767914"
  3.2721 +         sodipodi:role="line"
  3.2722 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2723 +         id="tspan3575">return from</tspan><tspan
  3.2724 +         id="tspan3577"
  3.2725 +         y="343.62958"
  3.2726 +         x="-39.767914"
  3.2727 +         sodipodi:role="line"
  3.2728 +         style="font-size:8px;text-align:center;text-anchor:middle">suspend</tspan></text>
  3.2729 +    <path
  3.2730 +       inkscape:connector-curvature="0"
  3.2731 +       id="path3579"
  3.2732 +       d="m 701.67453,400.02605 c -669.895225,0 -669.895225,0 -669.895225,0"
  3.2733 +       style="opacity:0.3125;fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.99999982, 2.99999982;stroke-dashoffset:0;marker-end:none" />
  3.2734 +    <text
  3.2735 +       sodipodi:linespacing="100%"
  3.2736 +       id="text3581"
  3.2737 +       y="377.42468"
  3.2738 +       x="667.38037"
  3.2739 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2740 +       xml:space="preserve"><tspan
  3.2741 +         id="tspan3583"
  3.2742 +         y="377.42468"
  3.2743 +         x="667.38037"
  3.2744 +         sodipodi:role="line"
  3.2745 +         style="text-align:center;text-anchor:middle;fill:#000080">PR primitive Fn</tspan><tspan
  3.2746 +         y="387.42468"
  3.2747 +         x="667.38037"
  3.2748 +         sodipodi:role="line"
  3.2749 +         style="text-align:center;text-anchor:middle;fill:#000080"
  3.2750 +         id="tspan3585">to send request</tspan></text>
  3.2751 +    <text
  3.2752 +       transform="matrix(0.99999036,0.00439064,-0.00439064,0.99999036,0,0)"
  3.2753 +       xml:space="preserve"
  3.2754 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:end;line-height:100%;writing-mode:lr-tb;text-anchor:end;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2755 +       x="258.57156"
  3.2756 +       y="402.12799"
  3.2757 +       id="text3587"
  3.2758 +       sodipodi:linespacing="100%"><tspan
  3.2759 +         style="font-size:8px;text-align:end;text-anchor:end"
  3.2760 +         sodipodi:role="line"
  3.2761 +         x="258.57156"
  3.2762 +         y="402.12799"
  3.2763 +         id="tspan3589">resume</tspan></text>
  3.2764 +    <path
  3.2765 +       inkscape:connector-curvature="0"
  3.2766 +       id="path3591"
  3.2767 +       d="m 205.05995,286.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.2768 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2769 +    <path
  3.2770 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2771 +       d="m 137.04459,336.02605 c 67.19099,0 67.19099,0 67.19099,0"
  3.2772 +       id="path3593"
  3.2773 +       inkscape:connector-curvature="0" />
  3.2774 +    <path
  3.2775 +       inkscape:connector-curvature="0"
  3.2776 +       id="path3595"
  3.2777 +       d="m 67.525286,310.02605 c 69.221574,0 69.221574,0 69.221574,0"
  3.2778 +       style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2779 +    <path
  3.2780 +       sodipodi:nodetypes="cc"
  3.2781 +       inkscape:connector-curvature="0"
  3.2782 +       id="path3597"
  3.2783 +       d="m 67.27805,297.85824 c 0,13.94668 0,13.94668 0,13.94668"
  3.2784 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2785 +    <path
  3.2786 +       inkscape:connector-curvature="0"
  3.2787 +       id="path3599"
  3.2788 +       d="m 137.04459,312.02605 c 67.19099,0 67.19099,0 67.19099,0"
  3.2789 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2790 +    <path
  3.2791 +       sodipodi:nodetypes="cc"
  3.2792 +       inkscape:connector-curvature="0"
  3.2793 +       id="path3601"
  3.2794 +       d="m 205.27805,312.63926 c 0,11.23237 0,11.23237 0,11.23237"
  3.2795 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2796 +    <path
  3.2797 +       style="fill:none;stroke:#000000;stroke-width:0.99999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2798 +       d="m 205.05995,324.02605 c -67.19099,0 -67.19099,0 -67.19099,0"
  3.2799 +       id="path3603"
  3.2800 +       inkscape:connector-curvature="0" />
  3.2801 +    <text
  3.2802 +       sodipodi:linespacing="100%"
  3.2803 +       id="text3605"
  3.2804 +       y="326.42468"
  3.2805 +       x="186.89299"
  3.2806 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2807 +       xml:space="preserve"
  3.2808 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"><tspan
  3.2809 +         id="tspan3607"
  3.2810 +         y="326.42468"
  3.2811 +         x="186.89299"
  3.2812 +         sodipodi:role="line"
  3.2813 +         style="font-size:7px;text-align:center;text-anchor:middle">call via Ptr to Dyn Lib Fn</tspan></text>
  3.2814 +    <path
  3.2815 +       sodipodi:nodetypes="cc"
  3.2816 +       inkscape:connector-curvature="0"
  3.2817 +       id="path3609"
  3.2818 +       d="m 137.27805,323.85824 c 0,11.43122 0,11.43122 0,11.43122"
  3.2819 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2820 +    <text
  3.2821 +       transform="matrix(0.99986317,0.01654229,-0.01654229,0.99986317,0,0)"
  3.2822 +       xml:space="preserve"
  3.2823 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2824 +       x="186.26445"
  3.2825 +       y="288.42987"
  3.2826 +       id="text3611"
  3.2827 +       sodipodi:linespacing="100%"><tspan
  3.2828 +         style="font-size:7px;text-align:center;text-anchor:middle"
  3.2829 +         sodipodi:role="line"
  3.2830 +         x="186.26445"
  3.2831 +         y="288.42987"
  3.2832 +         id="tspan3613">call via Ptr to Dyn Lib Fn</tspan></text>
  3.2833 +    <path
  3.2834 +       style="fill:none;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2835 +       d="m 465.78297,400.02605 c -193.92439,0 -193.92439,0 -193.92439,0"
  3.2836 +       id="path3615"
  3.2837 +       inkscape:connector-curvature="0" />
  3.2838 +    <path
  3.2839 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)"
  3.2840 +       d="m 466.19516,280.02605 c -193.92439,0 -193.92439,0 -193.92439,0"
  3.2841 +       id="path3617"
  3.2842 +       inkscape:connector-curvature="0" />
  3.2843 +    <text
  3.2844 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
  3.2845 +       xml:space="preserve"
  3.2846 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2847 +       x="35.877613"
  3.2848 +       y="497.93744"
  3.2849 +       id="text3619"
  3.2850 +       sodipodi:linespacing="100%"><tspan
  3.2851 +         id="tspan3621"
  3.2852 +         style="font-size:18px;text-align:center;text-anchor:middle"
  3.2853 +         sodipodi:role="line"
  3.2854 +         x="35.877613"
  3.2855 +         y="497.93744">core 1</tspan></text>
  3.2856 +    <path
  3.2857 +       sodipodi:nodetypes="cc"
  3.2858 +       inkscape:connector-curvature="0"
  3.2859 +       id="path3623"
  3.2860 +       d="m 205.27805,400.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.2861 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2862 +    <path
  3.2863 +       sodipodi:nodetypes="cc"
  3.2864 +       inkscape:connector-curvature="0"
  3.2865 +       id="path3625"
  3.2866 +       d="m 205.27805,400.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.2867 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2868 +    <path
  3.2869 +       sodipodi:nodetypes="cc"
  3.2870 +       inkscape:connector-curvature="0"
  3.2871 +       id="path3627"
  3.2872 +       d="m 205.27805,400.31985 c 0,7.35629 0,7.35629 0,7.35629"
  3.2873 +       style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" />
  3.2874 +    <text
  3.2875 +       transform="matrix(0.71848067,-0.69554692,0.69554692,0.71848067,0,0)"
  3.2876 +       xml:space="preserve"
  3.2877 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2878 +       x="-123.23382"
  3.2879 +       y="421.84821"
  3.2880 +       id="text3629"
  3.2881 +       sodipodi:linespacing="100%"><tspan
  3.2882 +         id="tspan3631"
  3.2883 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2884 +         sodipodi:role="line"
  3.2885 +         x="-123.23382"
  3.2886 +         y="421.84821">return from</tspan><tspan
  3.2887 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2888 +         sodipodi:role="line"
  3.2889 +         x="-123.23382"
  3.2890 +         y="429.84821"
  3.2891 +         id="tspan3633">suspend</tspan></text>
  3.2892 +    <path
  3.2893 +       inkscape:connector-curvature="0"
  3.2894 +       id="path3635"
  3.2895 +       d="m 271.03421,246.02605 c 193.92439,0 193.92439,0 193.92439,0"
  3.2896 +       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" />
  3.2897 +    <text
  3.2898 +       sodipodi:linespacing="100%"
  3.2899 +       id="text4289"
  3.2900 +       y="155.76575"
  3.2901 +       x="426.92542"
  3.2902 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  3.2903 +       xml:space="preserve"><tspan
  3.2904 +         y="155.76575"
  3.2905 +         x="426.92542"
  3.2906 +         id="tspan4291"
  3.2907 +         sodipodi:role="line"
  3.2908 +         style="text-align:center;text-anchor:middle">PR OS thread</tspan><tspan
  3.2909 +         y="164.20053"
  3.2910 +         x="426.92542"
  3.2911 +         sodipodi:role="line"
  3.2912 +         style="font-size:8px;text-align:center;text-anchor:middle"
  3.2913 +         id="tspan4295">(core 2)</tspan></text>
  3.2914 +    <g
  3.2915 +       transform="matrix(1,0,0,0.71130685,84,-77.47087)"
  3.2916 +       id="g4297">
  3.2917 +      <rect
  3.2918 +         y="313.58255"
  3.2919 +         x="310.40247"
  3.2920 +         height="52.250008"
  3.2921 +         width="65.276405"
  3.2922 +         id="rect4299"
  3.2923 +         style="fill:none;stroke:#000000;stroke-width:1.28076112;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
  3.2924 +    </g>
  3.2925 +  </g>
  3.2926 +</svg>