1.1 --- a/.hgignore	Fri May 04 09:44:14 2012 -0700
     1.2 +++ b/.hgignore	Wed May 16 06:32:02 2012 -0700
     1.3 @@ -1,8 +1,25 @@
     1.4 +nbproject
     1.5 +Makefile
     1.6 +build
     1.7 +dist
     1.8 +src/Default
     1.9 +src/.settings
    1.10 +src/.cproject
    1.11 +src/.project
    1.12 +.dep.inc
    1.13  
    1.14 -glob: *.o
    1.15 -glob: *.aux
    1.16 -glob: *.blg
    1.17 -glob: *.Backup
    1.18 -glob: *.bbl
    1.19 -glob: *.ddf
    1.20 -glob: *.Backup-*
    1.21 \ No newline at end of file
    1.22 +.cproject
    1.23 +.project
    1.24 +Debug
    1.25 +
    1.26 +# mercurial syntax as per: http://mercurial.selenic.com/wiki/.hgignore
    1.27 +.*\.Backup$
    1.28 +.*\.bb$
    1.29 +.*\.aux$
    1.30 +.*\.blg$
    1.31 +.*\.bbl$
    1.32 +.*\.ddf$
    1.33 +
    1.34 +# shell-style syntax, as per: http://www.selenic.com/mercurial/hgignore.5.html
    1.35 +syntax: glob
    1.36 +*.Backup*

     2.1 Binary file 0__Papers/Holistic_Model/Perf_Tune/latex/Holistic_Perf_Tuning.pdf has changed

     3.1 --- a/0__Papers/Holistic_Model/Perf_Tune/latex/Holistic_Perf_Tuning.tex	Fri May 04 09:44:14 2012 -0700
     3.2 +++ b/0__Papers/Holistic_Model/Perf_Tune/latex/Holistic_Perf_Tuning.tex	Wed May 16 06:32:02 2012 -0700
     3.3 @@ -283,9 +283,7 @@
     3.4  
     3.5  
     3.6  
     3.7 -\title{Performance Tuning Requires Integrating Aspects of Application, Runtime, Scheduling, and Hardware..\\
     3.8 -OR Performance Tuning Requires Connecting Application to Runtime, Scheduling, and Hardware\\
     3.9 -  OR Integrated Performance Tuning Using Semantic Information Collected by Instrumenting the Language Runtime}
    3.10 +\title{Integrated Performance Tuning Using Semantic Information Collected by Instrumenting the Language Runtime}
    3.11  
    3.12  \author{
    3.13          Nina Engelhardt\\
    3.14 @@ -322,149 +320,55 @@
    3.15  
    3.16  To fix this, a model is needed that provides the linkage to connect the measurements together.  It  should be in terms of scheduling decisions, having the units the decisions are made on, and the various sources of constraints on the decisions. 
    3.17  
    3.18 -Our contribution is such a model and framework. Using it, we generate views that indicate the schedulable units in the code, and all the constraints on scheduling them. These include constraints imposed by the application, the runtime implementation  and hardware details. 
    3.19 +Our contribution is such a model and framework. Using it, we generate views that indicate the schedulable units in the code, all the constraints on scheduling them, and connect them to application code. It includes constraints imposed by the application, the runtime implementation  and hardware details. 
    3.20  
    3.21 -The views connect the units to specific segments of code that compose the units, and connect each constraint on scheduling choice to the precise source of the constraint, within the code, hardware, or runtime. They separate resource usage into the categories: application work, non-overlapped communication (which results from scheduling decisions), and scheduling/runtime overhead. They also integrate parameter choices, within the code, that affect unit creation and constraints.
    3.22 +The connections made within the views encourage hypotheses for the cause of performance losses.  The views connect units to the specific segments of code that compose the units, and connect each constraint on scheduling choice to the precise source of the constraint, within the code, hardware, or runtime. They separate resource usage into the categories: application work, non-overlapped communication (which results from scheduling decisions), and scheduling/runtime overhead. They also integrate parameter choices, within the code, that affect unit creation and constraints. In visual form, the integration and connections have the effect of making areas of performance loss evident, and the causes of the loss clear. 
    3.23  
    3.24   
    3.25 -
    3.26 -
    3.27  We describe our model of computation, which relates the aspects of performance, and the instrumentation and visualization that are guided by that model and collect performance tuning information and link it together. The model and visualization are illustrated with a story line, which shows how they are used to performance tune a standard parallel application on a challenging multi-core system. 
    3.28  
    3.29  We start with background on performance tuning and previous approaches to tools in Section X. Then...
    3.30  
    3.31  
    3.32  \section{Background and Related Work}
    3.33 -Performance tuning follows the standard iterative approach. In each step of iteration, first measurements of performance are taken and compared to what was desired. Some mental model is  used to generate a hypothesis of why the measurement was less than desired. A mental model is then used to link the hypothesis to things within the programmer's control,  to suggest a change to make to the code. The modified code is then run again and the iteration-step repeats until the person doing the tuning is satisfied.
    3.34 +Performance tuning follows the standard pattern of iterating. In each iteration, first measurements of performance are taken and compared to what was desired. Some mental model is  used to generate a hypothesis of why the measurement was less than desired. A mental model is then used to link the hypothesis to things within the programmer's control,  to suggest a change to make to the code. The modified code is then run again and the iteration-step repeats until the person doing the tuning is satisfied.
    3.35  
    3.36 - 
    3.37  
    3.38 -For performance of parallel computation, more than just runtime measurements are important. Context for those measurements must also be gathered, which includes the characteristics of the units of work that are scheduled onto resources, as well as the  constraints on scheduling those. Then during the run, not only is hardware usage measured but also the scheduling decisions actually made and carried out.
    3.39 +For performance of parallel computation, more than just runtime measurements are important. Context for those measurements must also be gathered, which includes the characteristics of the units of work that are scheduled onto resources, as well as the  constraints on scheduling those. Then during the run, not only is hardware usage measured but also the scheduling decisions of why the hardware was used that way.
    3.40  
    3.41 +Based on this pattern of performance tuning, we divide other approaches to performance tuning into five categories and talk about them separately. First, the four classic approaches, then a newer breed of tool that is starting to emerge.
    3.42 +
    3.43 +\subsection{Thread-model based Approaches}
    3.44  Most of the older more established tools come from the threads world, and conceive of the application as a processor that performs actions, but don't include the concept of application-defined tasks nor constraints on them. This makes them unable to directly connect statistics they gather to application features.  The lack of connection forces the user to guess at what aspect of the code is responsible for observed performance.
    3.45  
    3.46  
    3.47 +
    3.48 +
    3.49  For example, Tau is a highly cited older system for performance tuning parallel applications, which is representative of thread-centric approaches. It integrates many data sources, and has rich displays. However its model was cores and memories and thread contexts, with actions taken on or by each. It had no well defined concept of scheduling, unit scheduled, nor constraints on scheduling those units. Hence, it had no view that integrated the parallelism-specific information at the heart of performance:  tasks, constraints on them, and scheduling choices.
    3.50  
    3.51 -Another highly cited classic performance tuning system is Paradyn[], which is representative of systems that model parallel computation as a collection of events. It is meant for applications that run for several days on multi-thousand node clusters. Its model of computation is based on events, both the timing of events and counts of events. It has a system for user-supplied instrumentation to collect event information and it has a hypotheses mechanism that protects the user from having to write custom code to test their hypotheses. However, the hypotheses are in terms of the timing and counts of events. not the parallel computation relevant information of units of scheduled work and the scheduling decisions made on those. 
    3.52 +\subsection{Application-intensive approach}
    3.53  
    3.54 +Another highly cited classic performance tuning system is Paradyn[], which is representative of systems that model parallel computation as a collection of events. It is meant for applications that run for several days on multi-thousand node clusters. Its model of computation is based on events, both the timing of events and counts of events. It has a system for user-supplied instrumentation to collect event information and it has a hypotheses mechanism that protects the user from having to write custom code to test their hypotheses. However, the hypotheses are in terms of the timing and counts of events, not the parallel computation relevant information of units of scheduled work and the scheduling decisions made on those. 
    3.55  
    3.56 +\subsection{Event driven approach}
    3.57  
    3.58  Paragraph also follows an event-based model, and represents the large collection of simpler tools that instrument the MPI library. It shows whether cores are busy, and indicates communication overhead, but lacks any features that tie the communication pattern realized to application code features, which are what is under programmer control. It also fails to show runtime overhead, and which portions of idle time are caused by runtime internal constraints.
    3.59  
    3.60  Paraver and Vampir are just painting tools that take event measurements and paint them on the screen.
    3.61  
    3.62 -Other approaches concentrate on performance counter data to identify hot-spots and potential false-sharing..  these suffer from the same lack of encompasing computation model, leaving the user to guess at what might be the cause of measured numbers.  The do a good job of saying that something might be wrong, bor a poor job of pointing to what is causing the problem, and hence leave the user baffled as to what to change in their code to get better performance.
    3.63 +\subsection{Performance-counter approaches}
    3.64 +Other approaches, such as VTune, POPI, and so forth concentrate on performance counter data to identify hot-spots and potential false-sharing.  These suffer from the same lack of encompassing computation model, leaving the user to guess at what might be the cause of measured numbers.  They do a good job of saying that something might be wrong, but a poor job of pointing to what is causing the problem, and hence leave the user baffled as to what to change in their code to get better performance.
    3.65  
    3.66 +\subsection{Newer approaches}
    3.67  The commonality among the classic approaches is the lack of a  model of parallel computation. 
    3.68  
    3.69  One difficulty faced by early tools is that  parallel applications written in MPI or threads effectively end up implementing a runtime system in the application code. In such a case, the units of work are implied in the code, and difficult for tools to recognize. Likewise, constraints on scheduling are enforced by the code, but never stated in any explicit form. 
    3.70  
    3.71 - These shortcomings are rectified by recent parallel languages such as CnC and StarSs, which cleanly identify tasks and constraints on them. The people who develop such a language also develop language-specific tools to go with it. 
    3.72 -
    3.73 -StarSs is clearly thinking about tasks, and even some about scheduling. but limited on scheduling -- can just place task into a queue, so can sort of manipulate scheduling but view doesn't really give all the constraints -- missing the runtime overhead, and missing the idle-time consequent from non-overlapped comm. 
    3.74 + These shortcomings are rectified by recent parallel languages such as CnC and StarSs, which cleanly identify tasks and the constraints on them. The people who develop such a language also develop language-specific tools to go with it. 
    3.75  
    3.76  The StarSs tool tries to simplify the view for the user. It doesn't give performance information directly, but instead identifies tasks and tells the user whether it thinks the task size is too small, just right, or   -- instead, it has a recommended task size, which is between too small, which has too much overhead and too big, which has too few tasks to load balance. It makes task a one color if too short, another if just right, and a third if too long.
    3.77  
    3.78  
    3.79 -
    3.80 -
    3.81 -==========
    3.82 -
    3.83 -MPI is also machine-based abstraction, that gives communication information, but doesn't have concept of constraints . Its sort of in-between..
    3.84 -
    3.85 -For communication, want two things: 1) idle-time on cores that is consequence from particular communication pattern, and in some cases 2) the energy due to the volume of communication. Both are consequence of the scheduling choices made.
    3.86 -
    3.87 -
    3.88 -
    3.89 -
    3.90 -=======================?
    3.91 -
    3.92 -Their structure information is either call graph or line of code. This is too limited for parallel performance. Need tasks and scheduling. More recent languages move towards task-based. This makes more of the needed structure information available. But we've not seen a complete or coherent presentation.
    3.93 -
    3.94 -=====
    3.95 - 
    3.96 - The value of a consequence graph is linking the size of boxes in it to the decisions made by the scheduler, which are represented by the shape. This lets a person visually link scheduling decisions to consequences.  The mind can quickly see empty areas, representing lost performance, and connect those to boxes and lines that cause the empty areas.  They then look up the code of each box and construct of each line, which gives them the code structures that caused the loss of performance.  This \emph{visual} linkage of undesirable effect back to \emph{cause in the code} is what sets our approach apart. 
    3.97 -The value of the UCC is visually linking cause of performance loss to options available for fixing it.
    3.98 -
    3.99 -===== 
   3.100 -
   3.101 -
   3.102 -
   3.103 -A survey of the most highly cited classic papers shows the commonality..
   3.104 -
   3.105 -===========
   3.106 -
   3.107 -
   3.108 -Use case: takes longer to create when more units -- that's because it mallocs all the matrix pieces, before it pairs any and starts the pair as work -- can see this from the graphs by linking the units to code they execute -- not the functions, specifically, but to the scheduled code-snippet. -- suggests changing the way divider works.
   3.109 -
   3.110 -Graph tells us the division code is bad -- works better when have 39 cores instead of 40 -- makes more pieces for 39 than for 40, and the end up fitting better onto the cores..
   3.111 -
   3.112 -Systematic way of adding auto-tuning to language -- using TSC for time measurements, and counters for cache miss.  Provides a framework that links the pieces together, and the UCC gives critical information useful in tuning.
   3.113 -
   3.114 -=======================
   3.115 -
   3.116 -Advantage: unlike statistical, it catches all "one off" and momentary serializations, which cause massive idling.
   3.117 -
   3.118 -Advantage: it is quick and easy to instrument a language that uses the same runtime abstraction (VMS) -- VMS was made to make language runtimes quick and easy, with low overhead (cite). VMS has the base instrumentation in it, language adds the types of constraints and the collection points. Each request to the runtime is in a separate (inlined) function, which makes adding the instrumentation simple.
   3.119 -
   3.120 -Advantage: instrument the language, NOT the application -- and yet it gives all the application-specific information from a run. -- others: Tau forces extensive instrumentation of application..  MPI based ones don't give the constraint info, nor what occupies time of core info..
   3.121 -
   3.122 -=======================
   3.123 -   
   3.124 -Expl of what is meant by "structure" info -- example where meas of runtime system showed that overhead of task creation took longer than task execution. Hypothesis was trivial: cause of lost performance is runtime overhead of creation is larger than work in a scheduled unit.  The plan to fix is to change the number of work-units created, by changing the parameter in the divider code.  Implementing this and re-executing showed that this source of performance loss was fixed by the change.
   3.125 -
   3.126 -The example shows that theory is part of hypothesis generation, because it required knowledge of the runtime and understanding that creation of a task is work performed as overhead inside the runtime.  The example also shows that generating the plan to fix required understanding the segment of code that divided work into tasks, and the relationship between parameters to that code and execution time of the resulting tasks.
   3.127 -
   3.128 -Hence, each step of performance debugging involves several aspects, including mental model of computation, application code, runtime implementation, scheduling choices, and hardware. In order to be effective, a tool used during performance tuning must be part of a complete model of computation that ties all aspects of the debugging/tuning steps together. Current tools fall short, both because they lack an encompassing model of computation, and because the tools are isolated from each other. Without integration, the user gets an incomplete picture of the computation and must resort to guesses either of where the problem lies or of what to do to fix it.
   3.129 -
   3.130 -
   3.131 -We introduce in this paper a model of computation that ties all aspects of performance together, along with instrumentation and visualization that is guided by the model and that links all relevant performance tuning information together. The model and visualization tools are illustrated with a story line, which shows how they are used to performance tune the standard matrix-multiply application on two multi-core systems. 
   3.132 -
   3.133 -Although we use standard visualization techniques [cite], our approach differs from previous work in both theoretical and practical aspects. The theory we  use is The Holistic Model of Parallel Computation, which ties together parallelism construct semantics with scheduling choices made during a run, and specific measurements made on the cores.  When put into practice, new kinds of measurements are taken, which complete the picture presented to the user, and each measurement is tied to a specific segment of code. The resulting combination not only identifies each source of performance loss, but ties it back to specific causes and suggests precise fixes, all of which is illustrated in our story line.
   3.134 -
   3.135 -[maybe some stuff about features and benefits of our approach: no app instrumentation, it's all inside language runtime, very low overhead, integrated with VMS-based functional debugging, and so on]
   3.136 -
   3.137 -
   3.138 -Table of features/benefits comparing us to others -- in related work section.
   3.139 --- language invariant.. concepts are lang independent, visualization is language independent
   3.140 --- instr lang once, then works across apps
   3.141 --- Integrate measurements with runtime implementation details
   3.142 --- Integrate measurements of one kind with those of a different kind
   3.143 --- Integrate measurements of core activity with code snippets
   3.144 --- Underlying theory ties measurements to each other, to runtime, and to hardware
   3.145 --- Has concept of unit of work / task
   3.146 --- Includes communication between cores
   3.147 --- Shows what each core spends each bit of time on
   3.148 --- Visual representation directly indicates areas of performance loss
   3.149 -
   3.150 -\begin{table*}[ht!]
   3.151 -\caption
   3.152 -{Comparison of Features and Benefits
   3.153 -}
   3.154 -\label{tabPersonDaysLang}
   3.155 -\begin{center}
   3.156 -\begin{tabular}{llll}
   3.157 -           & Tau & Paradyn & ASH \\
   3.158 - Has Underlying Parallel Computation Model &  &  & Y\\
   3.159 - Has concept of schedulable unit of work &  &  & Y\\
   3.160 - Visualization is Invariant Across Languages      &  &  & Y\\
   3.161 - Instrument Language only (not app code) &  &  & Y\\
   3.162 - Integrates Measurements into whole picture &  &  & Y\\
   3.163 - Visualizes all core activities &  &  & Y\\
   3.164 - Core's activities are integrated with scheduling decisions &  &  & Y\\
   3.165 - Visualizes runtime overhead &  &  & Y\\
   3.166 - Visualizes consequences of communications &  &  & Y\\
   3.167 - Visually suggests \emph{hypothesis} for performance loss &  &  & Y\\
   3.168 -  &  &  & Y\\
   3.169 -  &  &  & Y\\
   3.170 -  &  &  & Y\\
   3.171 -      L.O.C. &     470 & 290 & 310
   3.172 -  \end{tabular}
   3.173 -\end{center}
   3.174 -\end{table*}
   3.175 -
   3.176 -
   3.177  \section{Illustrative Story of Performance Tuning}
   3.178  
   3.179  
   3.180 @@ -524,14 +428,14 @@
   3.181  
   3.182  The application code includes a function that automatically divides the work into a number of tasks, based on the number of cores and a tuning parameter. It distributes the tasks across the cores in a round-robin fashion, and then waits for completion of the calculation before initiating shutdown. The answers produced by the tasks are collected by a ``receiver" and accumulated into the result matrix, whereupon it notifies the setup function of completion.
   3.183  
   3.184 -The language used is SSR, which is based on rendez-vous style send and receive operations made between virtual processors (VPs). It has commands for creating and destroying virtutal processors, and three kinds of send-receive paired operations. The first, \emph{send\_from\_to} specifies a specific sender and specific receiver. We use it to notify the VP dedicated to dividing the work (``creator VP'') that the VP collecting results (``results VP'') has received all sub-results, so it can output the final result. The second, \emph{send\_of\_type\_to}, specifies a specific receiver, but the sender is anonymous, which increases flexibility while maintaining some control over scope. This construct is used by the processors multiplying sub-matrices to send their sub-result to the results processor. The third kind, \emph{send\_of\_type}, only specifies the type, and so acts as a global communication channel; this is not used in our application. The language also includes a \emph{singleton} construct that designates a piece of code as to be executed only once, which we use to  rearrange and copy data to get better cache behavior. A given copy is shared by several virtual processors, but the copy only needs to be performed once. Also available are some miscellaneous performance constructs, such as the ability to force which core a virtual processor is assigned to, which is used in our example program.
   3.185 +The language used is SSR, which is based on rendez-vous style send and receive operations made between virtual processors (VPs). It has commands for creating and destroying virtutal processors, and three kinds of send-receive paired operations. The first, \emph{send\_from\_to} specifies a specific sender and specific receiver. We use it to notify the VP dedicated to dividing the work (``creator VP'') that the VP collecting results (``results VP'') has received all sub-results, so it can output the final result. The second, \emph{send\_of\_type\_to}, specifies a specific receiver, but the sender is anonymous, which increases flexibility while maintaining some control over scope. This construct is used by the processors multiplying sub-matrices to send their sub-result to the results processor. The third kind, \emph{send\_of\_type}, only specifies the type, and so acts as a global communication channel; this is not used in our application. The language also includes a \emph{singleton} construct that designates a piece of code as to be executed only once, which we use to  rearrange and copy data to get better cache behavior. A given copy is shared by several virtual processors on different cores, but the copy only needs to be performed once. Miscellaneous performance constructs are also available, such as one to force which core a virtual processor is assigned to, which is used in our example program.
   3.186  
   3.187  A note on terminology: the word ``task''  has acquired multiple  meanings in the literature, making it a vague term. We often use, instead, the term  ``work-unit'', which is defined precisely as the trace-segment performed on a core, between two successive scheduling events, plus the set of datums consumed by that trace segment. The word task often maps well onto this precise definition, and we use both words, but mean the precise work-unit definition when we say task.
   3.188  
   3.189 -A second visualization, seen in Figure X, depicts the constraints on the scheduling decisions, which limit the choices the runtime is allowed to make. We call this the Unit & Constraint Collection, or UCC. This shows only application-derived information. 
   3.190 +A second visualization, seen in Figure X, depicts the constraints on the scheduling decisions, which limit the choices the runtime is allowed to make. We call this the Unit \& Constraint Collection, or UCC. This shows only application-derived information. 
   3.191  
   3.192  \subsection{Walk through}
   3.193 -After functional debugging, the first run produces the visualization seen in Fig X. This is what we refer to as a scheduling consequence graph. It depicts all the scheduling operations performed by the runtime, along with the consequent usage of the cores. 
   3.194 +After functional debugging, the first run produces the visualization seen in Fig \ref{fig:figure1}. This is what we refer to as a scheduling consequence graph. It depicts all the scheduling operations performed by the runtime, along with the consequent usage of the cores. 
   3.195  
   3.196  Each column is associated with one core. 
   3.197  A blue vertical block represents the time the core spends doing the actual work of one work-unit, the height being proportional to the number of cycles taken for execution. Just above each, in light blue, is the runtime overhead spent on that work-unit, which can be broken into pieces representing acquisition of the lock on the shared semantic state, time spent performing the semantics of the parallelism construct, time spent deciding which ready task to execute next, and time spent switching from virtual processor, to the runtime, and back. 
   3.198 @@ -541,7 +445,7 @@
   3.199  
   3.200  Many different orderings could also have been validly chosen. Which scheduler choices are valid is determined by three kinds of constraints: the application code constraints, hardware constraints, and runtime implementation imposed constraints. 
   3.201  
   3.202 -Returning to Fig \ref{story}, the lines in red, orange, and green represent application-code constraints that each tie two work-units together. The color represents the  kind of constraint imposed by one kind of  parallelism construct. Red is creation of a virtual processor, green is many-to-one send\_of\_type\_to, and orange is the singleton construct. For better visibility, only constraints that cross cores are shown.
   3.203 +Returning to Fig \ref{story}, the lines in red, orange, and green represent application-code constraints that each tie two work-units together. The color represents the  kind of constraint imposed by one kind of  parallelism construct. Red is creation of a virtual processor, green is the many-to-one \texttt{send\_of\_type\_to}, and orange is the singleton construct. For better visibility, only constraints that cross cores are shown.
   3.204  
   3.205  The columns represent each core, and the vertical axis represents the progress of the program execution, starting from the top. This gives at a glance the total runtime (height), level of parallelism during the run (number of filled columns at a specific height), and processor utilization (blue-covered area vs. white). All figures are at the same scale, so they can be compared directly.
   3.206  
   3.207 @@ -809,6 +713,143 @@
   3.208  
   3.209  a lot of the time it's lists of measurements, or bar graphs, things like that -- over the whole application or by function -- forcing guessing of how it connects to -- if it tells you that this line creates a lot of level 2 cache misses, that doesn't tell you what the application is doing to cause this.. but when have whole UCC along with it, have context for the measurements -- puts the line of code into a framework -- it's necessary but not useful by itself -- it needs to be connected -- the unit information is more interesting than the line of code information -- line of code has only sequential meaning, missing scheduling connection -- need the scheduling behavior added -- need to know the unit of work that's causing problem, not the line of code -- unit provides a parallelism context, line of code does not.. unit provides an execution order and execution location, with implied communication -- line of code does not.
   3.210  
   3.211 +
   3.212 +
   3.213 +
   3.214 +======  Related work ========
   3.215 +
   3.216 +StarSs is clearly thinking about tasks, and even some about scheduling. but limited on scheduling -- can just place task into a queue, so can sort of manipulate scheduling but view doesn't really give all the constraints -- missing the runtime overhead, and missing the idle-time consequent from non-overlapped comm. 
   3.217 +
   3.218 +The PPoPP 12 paper and its ilk are fall into same category -- measurements without model to connect measurement to cause.
   3.219 +
   3.220 +-] MPI event based (Paragraph)
   3.221 +
   3.222 +-] Invasive Application Instrumentation approach (Paradyn)
   3.223 +
   3.224 +-] Thread-model based (Tau)
   3.225 +
   3.226 +-] profiling based (VTune, PPoPP 12 paper..)
   3.227 +
   3.228 +-] Task-based
   3.229 +(StarSs)
   3.230 +
   3.231 +?
   3.232 +
   3.233 +-] Instrument language runtime -- all applications inherit free
   3.234 +
   3.235 +-] Computation model backs it, connects effect back to cause
   3.236 +
   3.237 +-] Visual features indicate which hardware is idle
   3.238 +
   3.239 +-] Visual features indicate cause of idleness
   3.240 +
   3.241 +-] Low overhead instrumentation
   3.242 +
   3.243 +-] Task carries extra info that organizes --  connects disparate measurements -- helps connect effect back to cause
   3.244 +
   3.245 +-] Multiple languages easily reuse instrumentation 
   3.246 +
   3.247 +-] Visual features indicate langauge features from app-code
   3.248 +
   3.249 +?
   3.250 +
   3.251 +
   3.252 +==========
   3.253 +
   3.254 +MPI is also machine-based abstraction, that gives communication information, but doesn't have concept of constraints . Its sort of in-between..
   3.255 +
   3.256 +For communication, want two things: 1) idle-time on cores that is consequence from particular communication pattern, and in some cases 2) the energy due to the volume of communication. Both are consequence of the scheduling choices made.
   3.257 +
   3.258 +
   3.259 +
   3.260 +
   3.261 +=======================?
   3.262 +
   3.263 +Their structure information is either call graph or line of code. This is too limited for parallel performance. Need tasks and scheduling. More recent languages move towards task-based. This makes more of the needed structure information available. But we've not seen a complete or coherent presentation.
   3.264 +
   3.265 +=====
   3.266 + 
   3.267 + The value of a consequence graph is linking the size of boxes in it to the decisions made by the scheduler, which are represented by the shape. This lets a person visually link scheduling decisions to consequences.  The mind can quickly see empty areas, representing lost performance, and connect those to boxes and lines that cause the empty areas.  They then look up the code of each box and construct of each line, which gives them the code structures that caused the loss of performance.  This \emph{visual} linkage of undesirable effect back to \emph{cause in the code} is what sets our approach apart. 
   3.268 +The value of the UCC is visually linking cause of performance loss to options available for fixing it.
   3.269 +
   3.270 +===== 
   3.271 +
   3.272 +
   3.273 +
   3.274 +A survey of the most highly cited classic papers shows the commonality..
   3.275 +
   3.276 +===========
   3.277 +
   3.278 +
   3.279 +Use case: takes longer to create when more units -- that's because it mallocs all the matrix pieces, before it pairs any and starts the pair as work -- can see this from the graphs by linking the units to code they execute -- not the functions, specifically, but to the scheduled code-snippet. -- suggests changing the way divider works.
   3.280 +
   3.281 +Graph tells us the division code is bad -- works better when have 39 cores instead of 40 -- makes more pieces for 39 than for 40, and the end up fitting better onto the cores..
   3.282 +
   3.283 +Systematic way of adding auto-tuning to language -- using TSC for time measurements, and counters for cache miss.  Provides a framework that links the pieces together, and the UCC gives critical information useful in tuning.
   3.284 +
   3.285 +=======================
   3.286 +
   3.287 +Advantage: unlike statistical, it catches all "one off" and momentary serializations, which cause massive idling.
   3.288 +
   3.289 +Advantage: it is quick and easy to instrument a language that uses the same runtime abstraction (VMS) -- VMS was made to make language runtimes quick and easy, with low overhead (cite). VMS has the base instrumentation in it, language adds the types of constraints and the collection points. Each request to the runtime is in a separate (inlined) function, which makes adding the instrumentation simple.
   3.290 +
   3.291 +Advantage: instrument the language, NOT the application -- and yet it gives all the application-specific information from a run. -- others: Tau forces extensive instrumentation of application..  MPI based ones don't give the constraint info, nor what occupies time of core info..
   3.292 +
   3.293 +=======================
   3.294 +   
   3.295 +Expl of what is meant by "structure" info -- example where meas of runtime system showed that overhead of task creation took longer than task execution. Hypothesis was trivial: cause of lost performance is runtime overhead of creation is larger than work in a scheduled unit.  The plan to fix is to change the number of work-units created, by changing the parameter in the divider code.  Implementing this and re-executing showed that this source of performance loss was fixed by the change.
   3.296 +
   3.297 +The example shows that theory is part of hypothesis generation, because it required knowledge of the runtime and understanding that creation of a task is work performed as overhead inside the runtime.  The example also shows that generating the plan to fix required understanding the segment of code that divided work into tasks, and the relationship between parameters to that code and execution time of the resulting tasks.
   3.298 +
   3.299 +Hence, each step of performance debugging involves several aspects, including mental model of computation, application code, runtime implementation, scheduling choices, and hardware. In order to be effective, a tool used during performance tuning must be part of a complete model of computation that ties all aspects of the debugging/tuning steps together. Current tools fall short, both because they lack an encompassing model of computation, and because the tools are isolated from each other. Without integration, the user gets an incomplete picture of the computation and must resort to guesses either of where the problem lies or of what to do to fix it.
   3.300 +
   3.301 +
   3.302 +We introduce in this paper a model of computation that ties all aspects of performance together, along with instrumentation and visualization that is guided by the model and that links all relevant performance tuning information together. The model and visualization tools are illustrated with a story line, which shows how they are used to performance tune the standard matrix-multiply application on two multi-core systems. 
   3.303 +
   3.304 +Although we use standard visualization techniques [cite], our approach differs from previous work in both theoretical and practical aspects. The theory we  use is The Holistic Model of Parallel Computation, which ties together parallelism construct semantics with scheduling choices made during a run, and specific measurements made on the cores.  When put into practice, new kinds of measurements are taken, which complete the picture presented to the user, and each measurement is tied to a specific segment of code. The resulting combination not only identifies each source of performance loss, but ties it back to specific causes and suggests precise fixes, all of which is illustrated in our story line.
   3.305 +
   3.306 +[maybe some stuff about features and benefits of our approach: no app instrumentation, it's all inside language runtime, very low overhead, integrated with VMS-based functional debugging, and so on]
   3.307 +
   3.308 +
   3.309 +Table of features/benefits comparing us to others -- in related work section.
   3.310 +-- language invariant.. concepts are lang independent, visualization is language independent
   3.311 +-- instr lang once, then works across apps
   3.312 +-- Integrate measurements with runtime implementation details
   3.313 +-- Integrate measurements of one kind with those of a different kind
   3.314 +-- Integrate measurements of core activity with code snippets
   3.315 +-- Underlying theory ties measurements to each other, to runtime, and to hardware
   3.316 +-- Has concept of unit of work / task
   3.317 +-- Includes communication between cores
   3.318 +-- Shows what each core spends each bit of time on
   3.319 +-- Visual representation directly indicates areas of performance loss
   3.320 +
   3.321 +\begin{table*}[ht!]
   3.322 +\caption
   3.323 +{Comparison of Features and Benefits
   3.324 +}
   3.325 +\label{tabPersonDaysLang}
   3.326 +\begin{center}
   3.327 +\begin{tabular}{llll}
   3.328 +           & Tau & Paradyn & ASH \\
   3.329 + Has Underlying Parallel Computation Model &  &  & Y\\
   3.330 + Has concept of schedulable unit of work &  &  & Y\\
   3.331 + Visualization is Invariant Across Languages      &  &  & Y\\
   3.332 + Instrument Language only (not app code) &  &  & Y\\
   3.333 + Integrates Measurements into whole picture &  &  & Y\\
   3.334 + Visualizes all core activities &  &  & Y\\
   3.335 + Core's activities are integrated with scheduling decisions &  &  & Y\\
   3.336 + Visualizes runtime overhead &  &  & Y\\
   3.337 + Visualizes consequences of communications &  &  & Y\\
   3.338 + Visually suggests \emph{hypothesis} for performance loss &  &  & Y\\
   3.339 +  &  &  & Y\\
   3.340 +  &  &  & Y\\
   3.341 +  &  &  & Y\\
   3.342 +      L.O.C. &     470 & 290 & 310
   3.343 +  \end{tabular}
   3.344 +\end{center}
   3.345 +\end{table*}
   3.346 +
   3.347 +
   3.348  ===================
   3.349  
   3.350  Perf bounded by: the application-imposed constraints on scheduling units, the hardware-imposed constraints on scheduling them, and the runtime-implementation imposed constraints on scheduling them. 

     4.1 --- a/0__Papers/VMS/Universal/figures/control_flow.svg	Fri May 04 09:44:14 2012 -0700
     4.2 +++ b/0__Papers/VMS/Universal/figures/control_flow.svg	Wed May 16 06:32:02 2012 -0700
     4.3 @@ -9,8 +9,8 @@
     4.4     xmlns="http://www.w3.org/2000/svg"
     4.5     xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
     4.6     xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
     4.7 -   width="990"
     4.8 -   height="765"
     4.9 +   width="1488.189"
    4.10 +   height="1052.3622"
    4.11     id="svg2"
    4.12     sodipodi:version="0.32"
    4.13     inkscape:version="0.48.1 "
    4.14 @@ -520,9 +520,9 @@
    4.15       objecttolerance="10"
    4.16       inkscape:pageopacity="0.0"
    4.17       inkscape:pageshadow="2"
    4.18 -     inkscape:zoom="1.05"
    4.19 +     inkscape:zoom="0.97968464"
    4.20       inkscape:cx="457.74508"
    4.21 -     inkscape:cy="272.59162"
    4.22 +     inkscape:cy="264.16662"
    4.23       inkscape:document-units="px"
    4.24       inkscape:current-layer="layer1"
    4.25       showgrid="false"
    4.26 @@ -539,7 +539,7 @@
    4.27          <dc:format>image/svg+xml</dc:format>
    4.28          <dc:type
    4.29             rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
    4.30 -        <dc:title />
    4.31 +        <dc:title></dc:title>
    4.32        </cc:Work>
    4.33      </rdf:RDF>
    4.34    </metadata>
    4.35 @@ -547,7 +547,7 @@
    4.36       inkscape:label="Layer 1"
    4.37       inkscape:groupmode="layer"
    4.38       id="layer1"
    4.39 -     transform="translate(-3.5307846,441.70658)">
    4.40 +     transform="translate(-3.5307846,729.06884)">
    4.41      <text
    4.42         xml:space="preserve"
    4.43         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
    4.44 @@ -571,7 +571,7 @@
    4.45           x="368.25397"
    4.46           y="345.27225" /></text>
    4.47      <g
    4.48 -       transform="translate(-22.94669,-564)"
    4.49 +       transform="translate(-22.94669,-784)"
    4.50         id="g8065">
    4.51        <text
    4.52           xml:space="preserve"
    4.53 @@ -600,7 +600,7 @@
    4.54      </g>
    4.55      <g
    4.56         id="g11700"
    4.57 -       transform="translate(160,180)">
    4.58 +       transform="translate(160,-40)">
    4.59        <text
    4.60           xml:space="preserve"
    4.61           style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
    4.62 @@ -634,7 +634,7 @@
    4.63           style="fill:none;stroke:#000000;stroke-width:1.08748555;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
    4.64      </g>
    4.65      <g
    4.66 -       transform="translate(177.05331,-444)"
    4.67 +       transform="translate(177.05331,-664)"
    4.68         id="g8103">
    4.69        <text
    4.70           xml:space="preserve"
    4.71 @@ -663,7 +663,7 @@
    4.72      </g>
    4.73      <g
    4.74         id="g8165"
    4.75 -       transform="translate(177.05331,-384)">
    4.76 +       transform="translate(177.05331,-604)">
    4.77        <text
    4.78           sodipodi:linespacing="100%"
    4.79           id="text8167"
    4.80 @@ -690,7 +690,7 @@
    4.81           y="274.77512" />
    4.82      </g>
    4.83      <g
    4.84 -       transform="translate(437.05331,-444)"
    4.85 +       transform="translate(437.05331,-664)"
    4.86         id="g8173">
    4.87        <text
    4.88           xml:space="preserve"
    4.89 @@ -721,35 +721,35 @@
    4.90         sodipodi:nodetypes="cc"
    4.91         inkscape:connector-curvature="0"
    4.92         id="path8510"
    4.93 -       d="m 30.09217,-156.20795 134.75175,0"
    4.94 +       d="m 30.09217,-376.20795 134.75175,0"
    4.95         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#StopL);marker-end:url(#TriangleOutL)" />
    4.96      <text
    4.97         xml:space="preserve"
    4.98         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
    4.99         x="95.276283"
   4.100 -       y="-157.95343"
   4.101 +       y="-377.95343"
   4.102         id="text8067-3"
   4.103         sodipodi:linespacing="100%"><tspan
   4.104           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
   4.105           sodipodi:role="line"
   4.106           x="95.276283"
   4.107 -         y="-157.95343"
   4.108 +         y="-377.95343"
   4.109           id="tspan8071-6">Non-suspend end of work-unit</tspan></text>
   4.110      <path
   4.111         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#StopL);marker-end:url(#TriangleOutL)"
   4.112 -       d="m 30.09217,-276.20795 134.75175,0"
   4.113 +       d="m 30.09217,-496.20795 134.75175,0"
   4.114         id="path11002"
   4.115         inkscape:connector-curvature="0"
   4.116         sodipodi:nodetypes="cc" />
   4.117      <text
   4.118         sodipodi:linespacing="100%"
   4.119         id="text11004"
   4.120 -       y="-277.95343"
   4.121 +       y="-497.95343"
   4.122         x="95.430077"
   4.123         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.124         xml:space="preserve"><tspan
   4.125           id="tspan11006"
   4.126 -         y="-277.95343"
   4.127 +         y="-497.95343"
   4.128           x="95.430077"
   4.129           sodipodi:role="line"
   4.130           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1">Suspend at end of work-unit</tspan></text>
   4.131 @@ -757,17 +757,17 @@
   4.132         sodipodi:nodetypes="cc"
   4.133         inkscape:connector-curvature="0"
   4.134         id="path11472"
   4.135 -       d="m 399.56974,-144.00562 0,34.68367"
   4.136 +       d="m 399.56974,-364.00562 0,34.68367"
   4.137         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.138      <path
   4.139         sodipodi:nodetypes="cc"
   4.140         inkscape:connector-curvature="0"
   4.141         id="path11484"
   4.142 -       d="m 688.09217,-156.20795 22.13646,0"
   4.143 +       d="m 688.09217,-376.20795 22.13646,0"
   4.144         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.145      <path
   4.146         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
   4.147 -       d="m 768.09217,-156.20795 41.4633,0"
   4.148 +       d="m 768.09217,-376.20795 41.4633,0"
   4.149         id="path11518"
   4.150         inkscape:connector-curvature="0"
   4.151         sodipodi:nodetypes="cc" />
   4.152 @@ -787,58 +787,58 @@
   4.153         xml:space="preserve"
   4.154         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.155         x="729.43005"
   4.156 -       y="-121.95343"
   4.157 +       y="-341.95343"
   4.158         id="text11616"
   4.159         sodipodi:linespacing="100%"><tspan
   4.160           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.161           sodipodi:role="line"
   4.162           x="730.78357"
   4.163 -         y="-121.95343"
   4.164 +         y="-341.95343"
   4.165           id="tspan11618">     new </tspan><tspan
   4.166           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.167           sodipodi:role="line"
   4.168           x="729.43005"
   4.169 -         y="-112.95343"
   4.170 +         y="-332.95343"
   4.171           id="tspan12175">      work-unit</tspan><tspan
   4.172           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.173           sodipodi:role="line"
   4.174           x="729.43005"
   4.175 -         y="-103.95343"
   4.176 +         y="-323.95343"
   4.177           id="tspan11620">          is attached to</tspan><tspan
   4.178           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.179           sodipodi:role="line"
   4.180           x="729.43005"
   4.181 -         y="-94.95343"
   4.182 +         y="-314.95343"
   4.183           id="tspan11622">               a VP's context</tspan></text>
   4.184      <text
   4.185         sodipodi:linespacing="100%"
   4.186         id="text11624"
   4.187 -       y="-185.95343"
   4.188 +       y="-405.95343"
   4.189         x="787.43005"
   4.190         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.191         xml:space="preserve"><tspan
   4.192           id="tspan11626"
   4.193 -         y="-185.95343"
   4.194 +         y="-405.95343"
   4.195           x="787.43005"
   4.196           sodipodi:role="line"
   4.197           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">new work-</tspan><tspan
   4.198           id="tspan11630"
   4.199 -         y="-176.95343"
   4.200 +         y="-396.95343"
   4.201           x="787.43005"
   4.202           sodipodi:role="line"
   4.203           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">unit has</tspan><tspan
   4.204 -         y="-167.95343"
   4.205 +         y="-387.95343"
   4.206           x="787.43005"
   4.207           sodipodi:role="line"
   4.208           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.209           id="tspan12161">own local</tspan><tspan
   4.210 -         y="-158.95343"
   4.211 +         y="-378.95343"
   4.212           x="787.43005"
   4.213           sodipodi:role="line"
   4.214           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.215           id="tspan11634">context</tspan></text>
   4.216      <g
   4.217 -       transform="translate(617.05331,-463.31973)"
   4.218 +       transform="translate(617.05331,-683.31973)"
   4.219         id="g11656">
   4.220        <text
   4.221           xml:space="preserve"
   4.222 @@ -878,30 +878,30 @@
   4.223      <text
   4.224         sodipodi:linespacing="100%"
   4.225         id="text11718"
   4.226 -       y="-147.95343"
   4.227 +       y="-367.95343"
   4.228         x="95.276283"
   4.229         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.230         xml:space="preserve"><tspan
   4.231           id="tspan11720"
   4.232 -         y="-147.95343"
   4.233 +         y="-367.95343"
   4.234           x="95.276283"
   4.235           sodipodi:role="line"
   4.236           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1">which has attached context</tspan></text>
   4.237      <path
   4.238         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#StopL);marker-end:url(#TriangleOutL)"
   4.239 -       d="m 30.09217,-76.20795 134.75175,0"
   4.240 +       d="m 30.09217,-296.20795 134.75175,0"
   4.241         id="path11742"
   4.242         inkscape:connector-curvature="0"
   4.243         sodipodi:nodetypes="cc" />
   4.244      <text
   4.245         sodipodi:linespacing="100%"
   4.246         id="text11744"
   4.247 -       y="-77.95343"
   4.248 +       y="-297.95343"
   4.249         x="95.276283"
   4.250         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.251         xml:space="preserve"><tspan
   4.252           id="tspan11746"
   4.253 -         y="-77.95343"
   4.254 +         y="-297.95343"
   4.255           x="95.276283"
   4.256           sodipodi:role="line"
   4.257           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1">Non-suspend end of work-unit</tspan></text>
   4.258 @@ -909,17 +909,17 @@
   4.259         xml:space="preserve"
   4.260         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.261         x="95.276283"
   4.262 -       y="-67.95343"
   4.263 +       y="-287.95343"
   4.264         id="text11750"
   4.265         sodipodi:linespacing="100%"><tspan
   4.266           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
   4.267           sodipodi:role="line"
   4.268           x="95.276283"
   4.269 -         y="-67.95343"
   4.270 +         y="-287.95343"
   4.271           id="tspan11752">purely local (no ctxt attached)</tspan></text>
   4.272      <g
   4.273         id="g11754"
   4.274 -       transform="translate(-22.94669,-444)">
   4.275 +       transform="translate(-22.94669,-664)">
   4.276        <text
   4.277           sodipodi:linespacing="100%"
   4.278           id="text11756"
   4.279 @@ -949,51 +949,51 @@
   4.280         xml:space="preserve"
   4.281         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.282         x="29.276283"
   4.283 -       y="-129.95343"
   4.284 +       y="-349.95343"
   4.285         id="text11764"
   4.286         sodipodi:linespacing="100%"><tspan
   4.287           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.288           sodipodi:role="line"
   4.289           x="29.276283"
   4.290 -         y="-129.95343"
   4.291 +         y="-349.95343"
   4.292           id="tspan11766">CILK is example of this case,</tspan><tspan
   4.293           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.294           sodipodi:role="line"
   4.295           x="29.276283"
   4.296 -         y="-120.95343"
   4.297 +         y="-340.95343"
   4.298           id="tspan12203">when leaf child finishes</tspan></text>
   4.299      <text
   4.300         sodipodi:linespacing="100%"
   4.301         id="text11768"
   4.302 -       y="-49.95343"
   4.303 +       y="-269.95343"
   4.304         x="29.276283"
   4.305         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.306         xml:space="preserve"><tspan
   4.307           id="tspan11770"
   4.308 -         y="-49.95343"
   4.309 +         y="-269.95343"
   4.310           x="29.276283"
   4.311           sodipodi:role="line"
   4.312           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1">Dataflow is example of this case</tspan></text>
   4.313      <text
   4.314         sodipodi:linespacing="100%"
   4.315         id="text11772"
   4.316 -       y="-253.95343"
   4.317 +       y="-473.95343"
   4.318         x="29.276283"
   4.319         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.320         xml:space="preserve"><tspan
   4.321           id="tspan11774"
   4.322 -         y="-253.95343"
   4.323 +         y="-473.95343"
   4.324           x="29.276283"
   4.325           sodipodi:role="line"
   4.326           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1">pthread is example of this case,</tspan><tspan
   4.327 -         y="-244.95343"
   4.328 +         y="-464.95343"
   4.329           x="29.276283"
   4.330           sodipodi:role="line"
   4.331           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.332           id="tspan3625">as is Cilk when suspends on sync</tspan></text>
   4.333      <path
   4.334         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
   4.335 -       d="m 170.33352,-76.29839 100.31543,-78.14797"
   4.336 +       d="m 170.33352,-296.29839 100.31543,-78.14797"
   4.337         id="path11776"
   4.338         inkscape:connector-curvature="0"
   4.339         sodipodi:nodetypes="cc" />
   4.340 @@ -1001,61 +1001,61 @@
   4.341         sodipodi:nodetypes="cc"
   4.342         inkscape:connector-curvature="0"
   4.343         id="path12020"
   4.344 -       d="m 228.09216,-156.20795 41.06088,0"
   4.345 +       d="m 228.09216,-376.20795 41.06088,0"
   4.346         style="fill:none;stroke:#000000;stroke-width:0.99999994px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.347      <path
   4.348         sodipodi:nodetypes="cc"
   4.349         inkscape:connector-curvature="0"
   4.350         id="path12086"
   4.351 -       d="m 659.56974,-144.00562 0,24.4507"
   4.352 +       d="m 659.56974,-364.00562 0,24.4507"
   4.353         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.354      <text
   4.355         xml:space="preserve"
   4.356         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.357         x="661.43005"
   4.358 -       y="-135.95343"
   4.359 +       y="-355.95343"
   4.360         id="text12088"
   4.361         sodipodi:linespacing="100%"><tspan
   4.362           style="font-size:9px;text-align:start;text-anchor:start;fill:#008000;fill-opacity:1"
   4.363           sodipodi:role="line"
   4.364           x="661.43005"
   4.365 -         y="-135.95343"
   4.366 +         y="-355.95343"
   4.367           id="tspan12090">no work</tspan><tspan
   4.368           style="font-size:9px;text-align:start;text-anchor:start;fill:#008000;fill-opacity:1"
   4.369           sodipodi:role="line"
   4.370           x="661.43005"
   4.371 -         y="-126.95343"
   4.372 +         y="-346.95343"
   4.373           id="tspan12092">in Q</tspan></text>
   4.374      <path
   4.375         sodipodi:nodetypes="cc"
   4.376         inkscape:connector-curvature="0"
   4.377         id="path12155"
   4.378 -       d="M 737.20425,-133.48137 809.61651,-76.979"
   4.379 +       d="m 737.20425,-353.48137 72.41226,56.50237"
   4.380         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.381      <path
   4.382         sodipodi:nodetypes="cc"
   4.383         inkscape:connector-curvature="0"
   4.384         id="path12197"
   4.385 -       d="m 688.33352,-96.29839 23.17876,-59.17839"
   4.386 +       d="m 688.33352,-316.29839 23.17876,-59.17839"
   4.387         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.388      <text
   4.389         sodipodi:linespacing="100%"
   4.390         id="text12205"
   4.391 -       y="-192.5929"
   4.392 +       y="-412.5929"
   4.393         x="369.27625"
   4.394         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.395         xml:space="preserve"><tspan
   4.396           id="tspan12209"
   4.397 -         y="-192.5929"
   4.398 +         y="-412.5929"
   4.399           x="369.27625"
   4.400           sodipodi:role="line"
   4.401           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1">(in CILK, counts</tspan><tspan
   4.402 -         y="-183.5929"
   4.403 +         y="-403.5929"
   4.404           x="369.27625"
   4.405           sodipodi:role="line"
   4.406           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.407           id="tspan12307"> child ends, and</tspan><tspan
   4.408 -         y="-174.5929"
   4.409 +         y="-394.5929"
   4.410           x="369.27625"
   4.411           sodipodi:role="line"
   4.412           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.413 @@ -1064,33 +1064,33 @@
   4.414         xml:space="preserve"
   4.415         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.416         x="627.27625"
   4.417 -       y="-191.95343"
   4.418 +       y="-411.95343"
   4.419         id="text12213"
   4.420         sodipodi:linespacing="100%"><tspan
   4.421           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.422           sodipodi:role="line"
   4.423           x="627.27625"
   4.424 -         y="-191.95343"
   4.425 +         y="-411.95343"
   4.426           id="tspan12221">(for CILK, Q</tspan><tspan
   4.427           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.428           sodipodi:role="line"
   4.429           x="627.27625"
   4.430 -         y="-182.95343"
   4.431 +         y="-402.95343"
   4.432           id="tspan12305"> filled by async</tspan><tspan
   4.433           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.434           sodipodi:role="line"
   4.435           x="627.27625"
   4.436 -         y="-173.95343"
   4.437 +         y="-393.95343"
   4.438           id="tspan12219"> spawns)</tspan></text>
   4.439      <path
   4.440         sodipodi:nodetypes="cc"
   4.441         inkscape:connector-curvature="0"
   4.442         id="path12275"
   4.443 -       d="m 328.09217,-156.20795 41.44502,0"
   4.444 +       d="m 328.09217,-376.20795 41.44502,0"
   4.445         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.446      <g
   4.447         id="g12277"
   4.448 -       transform="translate(-280,180)">
   4.449 +       transform="translate(-280,-40)">
   4.450        <text
   4.451           sodipodi:linespacing="100%"
   4.452           id="text12279"
   4.453 @@ -1120,13 +1120,13 @@
   4.454      </g>
   4.455      <path
   4.456         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
   4.457 -       d="m 299.56974,-131.98903 0,122.094923"
   4.458 +       d="m 299.56974,-351.98903 0,122.09492"
   4.459         id="path12297"
   4.460         inkscape:connector-curvature="0"
   4.461         sodipodi:nodetypes="cc" />
   4.462      <g
   4.463         id="g12335"
   4.464 -       transform="translate(77.05331,-283.31973)">
   4.465 +       transform="translate(77.05331,-503.31973)">
   4.466        <text
   4.467           sodipodi:linespacing="100%"
   4.468           id="text12337"
   4.469 @@ -1148,22 +1148,22 @@
   4.470             y="303.39841"
   4.471             x="222.37003"
   4.472             sodipodi:role="line"
   4.473 -           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1">give them to</tspan><tspan
   4.474 -           id="tspan12345"
   4.475 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1">give to hdlrs.</tspan><tspan
   4.476             y="312.39841"
   4.477             x="222.37003"
   4.478             sodipodi:role="line"
   4.479 -           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1">constraint</tspan><tspan
   4.480 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
   4.481 +           id="tspan12400">check timers,</tspan><tspan
   4.482             y="321.39841"
   4.483             x="222.37003"
   4.484             sodipodi:role="line"
   4.485             style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
   4.486 -           id="tspan12398">updater and</tspan><tspan
   4.487 +           id="tspan3521">fire hdlr for</tspan><tspan
   4.488             y="330.39841"
   4.489             x="222.37003"
   4.490             sodipodi:role="line"
   4.491             style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
   4.492 -           id="tspan12400">Assigner</tspan></text>
   4.493 +           id="tspan3523">expired ones</tspan></text>
   4.494        <rect
   4.495           style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0"
   4.496           id="rect12347"
   4.497 @@ -1174,7 +1174,7 @@
   4.498      </g>
   4.499      <g
   4.500         id="g12349"
   4.501 -       transform="translate(617.05331,-383.31973)">
   4.502 +       transform="translate(617.05331,-603.31973)">
   4.503        <text
   4.504           sodipodi:linespacing="100%"
   4.505           id="text12351"
   4.506 @@ -1217,7 +1217,7 @@
   4.507      </g>
   4.508      <g
   4.509         id="g12404"
   4.510 -       transform="translate(17.05331,-184)">
   4.511 +       transform="translate(17.05331,-404)">
   4.512        <text
   4.513           sodipodi:linespacing="100%"
   4.514           id="text12406"
   4.515 @@ -1244,7 +1244,7 @@
   4.516           y="274.77512" />
   4.517      </g>
   4.518      <g
   4.519 -       transform="translate(97.05331,-124)"
   4.520 +       transform="translate(97.05331,-344)"
   4.521         id="g12418">
   4.522        <text
   4.523           xml:space="preserve"
   4.524 @@ -1275,11 +1275,11 @@
   4.525         sodipodi:nodetypes="cc"
   4.526         inkscape:connector-curvature="0"
   4.527         id="path12432"
   4.528 -       d="m 239.56974,115.99438 0,34.68367"
   4.529 +       d="m 239.56974,-104.00562 0,34.68367"
   4.530         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.531      <g
   4.532         id="g12434"
   4.533 -       transform="translate(-62.94669,-124)">
   4.534 +       transform="translate(-62.94669,-344)">
   4.535        <text
   4.536           sodipodi:linespacing="100%"
   4.537           id="text12436"
   4.538 @@ -1308,64 +1308,64 @@
   4.539      <text
   4.540         xml:space="preserve"
   4.541         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.542 -       x="353.27625"
   4.543 -       y="86.726822"
   4.544 +       x="273.27625"
   4.545 +       y="6.7268219"
   4.546         id="text12442"
   4.547         sodipodi:linespacing="100%"><tspan
   4.548           id="tspan12448"
   4.549           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.550           sodipodi:role="line"
   4.551 -         x="353.27625"
   4.552 -         y="86.726822">(in dataflow &amp; CILK,</tspan><tspan
   4.553 +         x="273.27625"
   4.554 +         y="6.7268219">(in dataflow &amp; CILK,</tspan><tspan
   4.555           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.556           sodipodi:role="line"
   4.557 -         x="353.27625"
   4.558 -         y="95.726822"
   4.559 +         x="273.27625"
   4.560 +         y="15.726822"
   4.561           id="tspan12454"> msgs from other cores</tspan><tspan
   4.562           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.563           sodipodi:role="line"
   4.564 -         x="353.27625"
   4.565 -         y="104.72682"
   4.566 -         id="tspan12456"> go to Assigner to ask</tspan><tspan
   4.567 +         x="273.27625"
   4.568 +         y="24.726822"
   4.569 +         id="tspan12482"> that ask for or push</tspan><tspan
   4.570           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.571           sodipodi:role="line"
   4.572 -         x="353.27625"
   4.573 -         y="113.72682"
   4.574 -         id="tspan12482"> for work and to push)</tspan></text>
   4.575 +         x="273.27625"
   4.576 +         y="33.726822"
   4.577 +         id="tspan3527"> work, go to assigner)</tspan></text>
   4.578      <text
   4.579         sodipodi:linespacing="100%"
   4.580         id="text12460"
   4.581 -       y="190.72681"
   4.582 -       x="359.27625"
   4.583 +       y="6.7268066"
   4.584 +       x="369.27625"
   4.585         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.586         xml:space="preserve"><tspan
   4.587           id="tspan12466"
   4.588 -         y="190.72681"
   4.589 -         x="359.27625"
   4.590 +         y="6.7268066"
   4.591 +         x="369.27625"
   4.592           sodipodi:role="line"
   4.593           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1">(in dataflow &amp; CILK,</tspan><tspan
   4.594 -         y="199.72681"
   4.595 -         x="359.27625"
   4.596 +         y="15.726807"
   4.597 +         x="369.27625"
   4.598           sodipodi:role="line"
   4.599           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.600           id="tspan12472"> push work to other</tspan><tspan
   4.601 -         y="208.72681"
   4.602 -         x="359.27625"
   4.603 +         y="24.726807"
   4.604 +         x="369.27625"
   4.605           sodipodi:role="line"
   4.606           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.607           id="tspan12488"> cores via msgs -- </tspan><tspan
   4.608 -         y="217.72681"
   4.609 -         x="359.27625"
   4.610 +         y="33.726807"
   4.611 +         x="369.27625"
   4.612           sodipodi:role="line"
   4.613           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.614           id="tspan3973"> remember </tspan><tspan
   4.615 -         y="226.72681"
   4.616 -         x="359.27625"
   4.617 +         y="42.726807"
   4.618 +         x="369.27625"
   4.619           sodipodi:role="line"
   4.620           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.621           id="tspan3977"> constraints that</tspan><tspan
   4.622 -         y="235.72681"
   4.623 -         x="359.27625"
   4.624 +         y="51.726807"
   4.625 +         x="369.27625"
   4.626           sodipodi:role="line"
   4.627           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.628           id="tspan12642"> cross cores)</tspan></text>
   4.629 @@ -1373,11 +1373,11 @@
   4.630         sodipodi:nodetypes="cc"
   4.631         inkscape:connector-curvature="0"
   4.632         id="path12490"
   4.633 -       d="M 328.33352,16.826612 369.78149,-154.44636"
   4.634 +       d="m 328.33352,-203.17339 41.44797,-171.27297"
   4.635         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.636      <g
   4.637         id="g12506"
   4.638 -       transform="translate(97.05331,-184)">
   4.639 +       transform="translate(97.05331,-404)">
   4.640        <text
   4.641           sodipodi:linespacing="100%"
   4.642           id="text12508"
   4.643 @@ -1403,43 +1403,15 @@
   4.644           x="193.78287"
   4.645           y="274.77512" />
   4.646      </g>
   4.647 -    <g
   4.648 -       transform="translate(177.05331,-124)"
   4.649 -       id="g12516">
   4.650 -      <text
   4.651 -         xml:space="preserve"
   4.652 -         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.653 -         x="222.25137"
   4.654 -         y="285.36765"
   4.655 -         id="text12518"
   4.656 -         sodipodi:linespacing="100%"><tspan
   4.657 -           id="tspan12520"
   4.658 -           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
   4.659 -           sodipodi:role="line"
   4.660 -           x="222.25137"
   4.661 -           y="285.36765">send work-</tspan><tspan
   4.662 -           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
   4.663 -           sodipodi:role="line"
   4.664 -           x="222.25137"
   4.665 -           y="294.36765"
   4.666 -           id="tspan12780">push msg</tspan></text>
   4.667 -      <rect
   4.668 -         y="274.77512"
   4.669 -         x="193.78287"
   4.670 -         height="25.544432"
   4.671 -         width="57.486332"
   4.672 -         id="rect12522"
   4.673 -         style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
   4.674 -    </g>
   4.675      <path
   4.676         sodipodi:nodetypes="cc"
   4.677         inkscape:connector-curvature="0"
   4.678         id="path12534"
   4.679 -       d="m 319.56974,115.99438 0,34.68367"
   4.680 +       d="m 319.56974,-104.00562 0,34.68367"
   4.681         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.682      <path
   4.683         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
   4.684 -       d="M 297.91051,54.87914 244.22867,90.53671"
   4.685 +       d="m 297.91051,-165.12086 -53.68184,35.65757"
   4.686         id="path12544"
   4.687         inkscape:connector-curvature="0"
   4.688         sodipodi:nodetypes="cc" />
   4.689 @@ -1447,10 +1419,10 @@
   4.690         sodipodi:nodetypes="cc"
   4.691         inkscape:connector-curvature="0"
   4.692         id="path12550"
   4.693 -       d="m 239.95132,116.23968 -77.23827,34.97636"
   4.694 +       d="m 239.95132,-103.76032 -77.23827,34.97636"
   4.695         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.696      <g
   4.697 -       transform="translate(257.05331,-384)"
   4.698 +       transform="translate(257.05331,-604)"
   4.699         id="g12552">
   4.700        <text
   4.701           xml:space="preserve"
   4.702 @@ -1481,26 +1453,26 @@
   4.703         sodipodi:nodetypes="cc"
   4.704         inkscape:connector-curvature="0"
   4.705         id="path12562"
   4.706 -       d="M 303.63081,54.87914 318.29909,89.8607"
   4.707 +       d="m 303.63081,-165.12086 14.66828,34.98156"
   4.708         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.709      <text
   4.710         sodipodi:linespacing="100%"
   4.711         id="text12564"
   4.712 -       y="-131.95343"
   4.713 +       y="-351.95343"
   4.714         x="371.43005"
   4.715         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.716         xml:space="preserve"><tspan
   4.717           id="tspan12568"
   4.718 -         y="-131.95343"
   4.719 +         y="-351.95343"
   4.720           x="371.43005"
   4.721           sodipodi:role="line"
   4.722           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">   done</tspan><tspan
   4.723 -         y="-122.95343"
   4.724 +         y="-342.95343"
   4.725           x="371.43005"
   4.726           sodipodi:role="line"
   4.727           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.728           id="tspan12598"> with</tspan><tspan
   4.729 -         y="-113.95343"
   4.730 +         y="-333.95343"
   4.731           x="371.43005"
   4.732           sodipodi:role="line"
   4.733           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.734 @@ -1509,27 +1481,27 @@
   4.735         xml:space="preserve"
   4.736         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.737         x="335.43005"
   4.738 -       y="-159.95343"
   4.739 +       y="-379.95343"
   4.740         id="text12572"
   4.741         sodipodi:linespacing="100%"><tspan
   4.742           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.743           sodipodi:role="line"
   4.744           x="335.43005"
   4.745 -         y="-159.95343"
   4.746 +         y="-379.95343"
   4.747           id="tspan12574">no</tspan></text>
   4.748      <text
   4.749         sodipodi:linespacing="100%"
   4.750         id="text12576"
   4.751 -       y="-123.95343"
   4.752 +       y="-343.95343"
   4.753         x="309.43005"
   4.754         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.755         xml:space="preserve"><tspan
   4.756           id="tspan12578"
   4.757 -         y="-123.95343"
   4.758 +         y="-343.95343"
   4.759           x="309.43005"
   4.760           sodipodi:role="line"
   4.761           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">yes,</tspan><tspan
   4.762 -         y="-114.95343"
   4.763 +         y="-334.95343"
   4.764           x="309.43005"
   4.765           sodipodi:role="line"
   4.766           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.767 @@ -1538,82 +1510,82 @@
   4.768         xml:space="preserve"
   4.769         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.770         x="251.43005"
   4.771 -       y="68.04657"
   4.772 +       y="-151.95343"
   4.773         id="text12582"
   4.774         sodipodi:linespacing="100%"><tspan
   4.775           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.776           sodipodi:role="line"
   4.777           x="251.43005"
   4.778 -         y="68.04657"
   4.779 +         y="-151.95343"
   4.780           id="tspan12584">  constraint</tspan><tspan
   4.781           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.782           sodipodi:role="line"
   4.783           x="251.43005"
   4.784 -         y="77.04657"
   4.785 +         y="-142.95343"
   4.786           id="tspan12586">msgs</tspan></text>
   4.787      <text
   4.788         sodipodi:linespacing="100%"
   4.789         id="text12592"
   4.790 -       y="68.04657"
   4.791 +       y="-151.95343"
   4.792         x="323.43005"
   4.793         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.794         xml:space="preserve"><tspan
   4.795           id="tspan12594"
   4.796 -         y="68.04657"
   4.797 +         y="-151.95343"
   4.798           x="323.43005"
   4.799           sodipodi:role="line"
   4.800           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">  Assigner</tspan><tspan
   4.801           id="tspan12596"
   4.802 -         y="77.04657"
   4.803 +         y="-142.95343"
   4.804           x="323.43005"
   4.805           sodipodi:role="line"
   4.806           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">msgs</tspan></text>
   4.807      <text
   4.808         sodipodi:linespacing="100%"
   4.809         id="text12602"
   4.810 -       y="84.726822"
   4.811 -       x="105.27625"
   4.812 +       y="-29.273178"
   4.813 +       x="129.27625"
   4.814         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.815         xml:space="preserve"><tspan
   4.816           id="tspan12606"
   4.817 -         y="84.726822"
   4.818 -         x="105.27625"
   4.819 +         y="-29.273178"
   4.820 +         x="129.27625"
   4.821           sodipodi:role="line"
   4.822           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1">(in CILK, completion of</tspan><tspan
   4.823 -         y="93.726822"
   4.824 -         x="105.27625"
   4.825 +         y="-20.273178"
   4.826 +         x="129.27625"
   4.827           sodipodi:role="line"
   4.828           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.829           id="tspan12614"> child on remote core</tspan><tspan
   4.830           id="tspan12610"
   4.831 -         y="102.72682"
   4.832 -         x="105.27625"
   4.833 +         y="-11.273178"
   4.834 +         x="129.27625"
   4.835           sodipodi:role="line"
   4.836           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"> notifies parent's core.</tspan><tspan
   4.837 -         y="111.72682"
   4.838 -         x="105.27625"
   4.839 +         y="-2.2731781"
   4.840 +         x="129.27625"
   4.841           sodipodi:role="line"
   4.842           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.843           id="tspan12624"> In dataflow, remote pro-</tspan><tspan
   4.844 -         y="120.72682"
   4.845 -         x="105.27625"
   4.846 +         y="6.7268219"
   4.847 +         x="129.27625"
   4.848           sodipodi:role="line"
   4.849           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.850           id="tspan12626"> pendent sends data to</tspan><tspan
   4.851 -         y="129.72682"
   4.852 -         x="105.27625"
   4.853 +         y="15.726822"
   4.854 +         x="129.27625"
   4.855           sodipodi:role="line"
   4.856           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
   4.857           id="tspan12634"> dependents' cores)</tspan></text>
   4.858      <path
   4.859         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
   4.860 -       d="m 401.6308,-143.12086 74.44669,32.98177"
   4.861 +       d="m 401.6308,-363.12086 74.44669,32.98177"
   4.862         id="path12658"
   4.863         inkscape:connector-curvature="0"
   4.864         sodipodi:nodetypes="cc" />
   4.865      <g
   4.866         id="g12764"
   4.867 -       transform="translate(437.05331,-264)">
   4.868 +       transform="translate(437.05331,-484)">
   4.869        <text
   4.870           sodipodi:linespacing="100%"
   4.871           id="text12766"
   4.872 @@ -1643,48 +1615,48 @@
   4.873         xml:space="preserve"
   4.874         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.875         x="299.43005"
   4.876 -       y="126.04657"
   4.877 +       y="-93.95343"
   4.878         id="text12816"
   4.879         sodipodi:linespacing="100%"><tspan
   4.880           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.881           sodipodi:role="line"
   4.882           x="299.43005"
   4.883 -         y="126.04657"
   4.884 +         y="-93.95343"
   4.885           id="tspan12818">  receive</tspan><tspan
   4.886           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.887           sodipodi:role="line"
   4.888           x="300.78357"
   4.889 -         y="135.04657"
   4.890 +         y="-84.95343"
   4.891           id="tspan12820">work-push   </tspan><tspan
   4.892           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.893           sodipodi:role="line"
   4.894           x="299.43005"
   4.895 -         y="144.04657"
   4.896 +         y="-75.95343"
   4.897           id="tspan12822">       msg</tspan></text>
   4.898      <path
   4.899         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
   4.900 -       d="m 321.6308,116.87914 74.44669,32.98177"
   4.901 +       d="m 321.6308,-103.12086 74.44669,32.98177"
   4.902         id="path12824"
   4.903         inkscape:connector-curvature="0"
   4.904         sodipodi:nodetypes="cc" />
   4.905      <text
   4.906         sodipodi:linespacing="100%"
   4.907         id="text12826"
   4.908 -       y="126.04657"
   4.909 +       y="-93.95343"
   4.910         x="379.43005"
   4.911         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.912         xml:space="preserve"><tspan
   4.913           id="tspan12828"
   4.914 -         y="126.04657"
   4.915 +         y="-93.95343"
   4.916           x="379.43005"
   4.917           sodipodi:role="line"
   4.918           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">  receive need-work</tspan><tspan
   4.919           id="tspan12832"
   4.920 -         y="135.04657"
   4.921 +         y="-84.95343"
   4.922           x="379.43005"
   4.923           sodipodi:role="line"
   4.924           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">                msg &amp; have work</tspan><tspan
   4.925 -         y="144.04657"
   4.926 +         y="-75.95343"
   4.927           x="379.43005"
   4.928           sodipodi:role="line"
   4.929           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.930 @@ -1692,18 +1664,18 @@
   4.931      <text
   4.932         sodipodi:linespacing="100%"
   4.933         id="text12914"
   4.934 -       y="-61.95343"
   4.935 +       y="-281.95343"
   4.936         x="678.38245"
   4.937         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
   4.938         xml:space="preserve"><tspan
   4.939 -         y="-61.95343"
   4.940 +         y="-281.95343"
   4.941           x="678.38245"
   4.942           sodipodi:role="line"
   4.943           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
   4.944           id="tspan12946">No work</tspan></text>
   4.945      <g
   4.946         id="g3699"
   4.947 -       transform="translate(717.05331,-444)">
   4.948 +       transform="translate(717.05331,-664)">
   4.949        <text
   4.950           sodipodi:linespacing="100%"
   4.951           id="text3701"
   4.952 @@ -1731,13 +1703,13 @@
   4.953      </g>
   4.954      <path
   4.955         style="fill:none;stroke:#000000;stroke-width:0.99999994px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
   4.956 -       d="m 868.0922,-156.20795 41.0608,0"
   4.957 +       d="m 868.0922,-376.20795 41.0608,0"
   4.958         id="path3709"
   4.959         inkscape:connector-curvature="0"
   4.960         sodipodi:nodetypes="cc" />
   4.961      <path
   4.962         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
   4.963 -       d="m 868.33356,-76.29838 40.86584,-76.648"
   4.964 +       d="m 868.33356,-296.29838 40.86584,-76.648"
   4.965         id="path3715"
   4.966         inkscape:connector-curvature="0"
   4.967         sodipodi:nodetypes="cc" />
   4.968 @@ -1745,11 +1717,11 @@
   4.969         sodipodi:nodetypes="cc"
   4.970         inkscape:connector-curvature="0"
   4.971         id="path3769"
   4.972 -       d="m 428.86861,-156.20795 200.12395,0"
   4.973 +       d="m 428.86861,-376.20795 200.12395,0"
   4.974         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
   4.975      <g
   4.976         id="g3775"
   4.977 -       transform="translate(297.05331,-564)">
   4.978 +       transform="translate(297.05331,-784)">
   4.979        <text
   4.980           sodipodi:linespacing="100%"
   4.981           id="text3777"
   4.982 @@ -1772,7 +1744,7 @@
   4.983      </g>
   4.984      <g
   4.985         id="g3789"
   4.986 -       transform="translate(177.05331,-564)">
   4.987 +       transform="translate(177.05331,-784)">
   4.988        <text
   4.989           sodipodi:linespacing="100%"
   4.990           id="text3791"
   4.991 @@ -1795,12 +1767,12 @@
   4.992      </g>
   4.993      <path
   4.994         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
   4.995 -       d="m 428.09218,-276.20795 60.12116,0"
   4.996 +       d="m 428.09218,-496.20795 60.12116,0"
   4.997         id="path3799"
   4.998         inkscape:connector-curvature="0"
   4.999         sodipodi:nodetypes="cc" />
  4.1000      <g
  4.1001 -       transform="translate(-300,60)"
  4.1002 +       transform="translate(-300,-160)"
  4.1003         id="g3801">
  4.1004        <text
  4.1005           xml:space="preserve"
  4.1006 @@ -1831,7 +1803,7 @@
  4.1007      </g>
  4.1008      <path
  4.1009         style="fill:none;stroke:#000000;stroke-width:0.99999994px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1010 -       d="m 228.09216,-276.20795 21.72127,0"
  4.1011 +       d="m 228.09216,-496.20795 21.72127,0"
  4.1012         id="path3811"
  4.1013         inkscape:connector-curvature="0"
  4.1014         sodipodi:nodetypes="cc" />
  4.1015 @@ -1839,40 +1811,40 @@
  4.1016         sodipodi:nodetypes="cc"
  4.1017         inkscape:connector-curvature="0"
  4.1018         id="path3813"
  4.1019 -       d="m 307.68401,-276.20795 61.46905,0"
  4.1020 +       d="m 307.68401,-496.20795 61.46905,0"
  4.1021         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1022      <text
  4.1023         xml:space="preserve"
  4.1024         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1025         x="289.43005"
  4.1026 -       y="-243.95343"
  4.1027 +       y="-463.95343"
  4.1028         id="text3815"
  4.1029         sodipodi:linespacing="100%"><tspan
  4.1030           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1031           sodipodi:role="line"
  4.1032           x="289.43005"
  4.1033 -         y="-243.95343"
  4.1034 +         y="-463.95343"
  4.1035           id="tspan3817">yes,</tspan><tspan
  4.1036           id="tspan3819"
  4.1037           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1038           sodipodi:role="line"
  4.1039           x="289.43005"
  4.1040 -         y="-234.95343">chk</tspan></text>
  4.1041 +         y="-454.95343">chk</tspan></text>
  4.1042      <path
  4.1043         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1044 -       d="m 328.33352,22.4516 41.44797,-295.744"
  4.1045 +       d="m 328.33352,-197.5484 41.44797,-295.744"
  4.1046         id="path3823"
  4.1047         inkscape:connector-curvature="0"
  4.1048         sodipodi:nodetypes="cc" />
  4.1049      <text
  4.1050         sodipodi:linespacing="100%"
  4.1051         id="text3825"
  4.1052 -       y="-279.95343"
  4.1053 +       y="-499.95343"
  4.1054         x="315.43005"
  4.1055         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1056         xml:space="preserve"><tspan
  4.1057           id="tspan3827"
  4.1058 -         y="-279.95343"
  4.1059 +         y="-499.95343"
  4.1060           x="315.43005"
  4.1061           sodipodi:role="line"
  4.1062           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">no</tspan></text>
  4.1063 @@ -1880,26 +1852,26 @@
  4.1064         xml:space="preserve"
  4.1065         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1066         x="353.43005"
  4.1067 -       y="-253.95343"
  4.1068 +       y="-473.95343"
  4.1069         id="text3829"
  4.1070         sodipodi:linespacing="100%"><tspan
  4.1071           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1072           sodipodi:role="line"
  4.1073           x="353.43005"
  4.1074 -         y="-253.95343"
  4.1075 +         y="-473.95343"
  4.1076           id="tspan3831">   done</tspan><tspan
  4.1077           id="tspan3833"
  4.1078           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1079           sodipodi:role="line"
  4.1080           x="353.43005"
  4.1081 -         y="-244.95343">  with</tspan><tspan
  4.1082 +         y="-464.95343">  with</tspan><tspan
  4.1083           id="tspan3835"
  4.1084           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1085           sodipodi:role="line"
  4.1086           x="353.43005"
  4.1087 -         y="-235.95343">msgs</tspan></text>
  4.1088 +         y="-455.95343">msgs</tspan></text>
  4.1089      <g
  4.1090 -       transform="translate(97.05331,-123.31973)"
  4.1091 +       transform="translate(97.05331,-343.31973)"
  4.1092         id="g3837">
  4.1093        <text
  4.1094           xml:space="preserve"
  4.1095 @@ -1938,7 +1910,7 @@
  4.1096      </g>
  4.1097      <g
  4.1098         id="g3851"
  4.1099 -       transform="translate(177.05331,-524)">
  4.1100 +       transform="translate(177.05331,-744)">
  4.1101        <text
  4.1102           sodipodi:linespacing="100%"
  4.1103           id="text3853"
  4.1104 @@ -1966,12 +1938,12 @@
  4.1105      </g>
  4.1106      <path
  4.1107         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1108 -       d="m 399.56974,-264.00562 0,12.75587"
  4.1109 +       d="m 399.56974,-484.00562 0,12.75587"
  4.1110         id="path3863"
  4.1111         inkscape:connector-curvature="0"
  4.1112         sodipodi:nodetypes="cc" />
  4.1113      <g
  4.1114 -       transform="translate(257.05331,-524)"
  4.1115 +       transform="translate(257.05331,-744)"
  4.1116         id="g3865">
  4.1117        <text
  4.1118           xml:space="preserve"
  4.1119 @@ -1999,7 +1971,7 @@
  4.1120           style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
  4.1121      </g>
  4.1122      <g
  4.1123 -       transform="translate(317.05331,-524)"
  4.1124 +       transform="translate(317.05331,-744)"
  4.1125         id="g3875">
  4.1126        <text
  4.1127           xml:space="preserve"
  4.1128 @@ -2030,29 +2002,29 @@
  4.1129         sodipodi:nodetypes="cc"
  4.1130         inkscape:connector-curvature="0"
  4.1131         id="path3887"
  4.1132 -       d="m 519.99814,-263.12085 17.68767,13.95011"
  4.1133 +       d="m 519.99814,-483.12085 17.68767,13.95011"
  4.1134         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1135      <path
  4.1136         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1137 -       d="m 516.1523,-263.12085 -35.11392,13.95011"
  4.1138 +       d="m 516.1523,-483.12085 -35.11392,13.95011"
  4.1139         id="path3889"
  4.1140         inkscape:connector-curvature="0"
  4.1141         sodipodi:nodetypes="cc" />
  4.1142      <path
  4.1143         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1144 -       d="m 548.6784,-275.12084 80.39811,116.64706"
  4.1145 +       d="m 548.6784,-495.12084 80.39811,116.64706"
  4.1146         id="path3891"
  4.1147         inkscape:connector-curvature="0"
  4.1148         sodipodi:nodetypes="cc" />
  4.1149      <path
  4.1150         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1151 -       d="m 737.20425,-178.67186 72.41226,-56.50237"
  4.1152 +       d="m 737.20425,-398.67186 72.41226,-56.50237"
  4.1153         id="path3893"
  4.1154         inkscape:connector-curvature="0"
  4.1155         sodipodi:nodetypes="cc" />
  4.1156      <g
  4.1157         id="g3895"
  4.1158 -       transform="translate(617.05331,-543.31973)">
  4.1159 +       transform="translate(617.05331,-763.31973)">
  4.1160        <text
  4.1161           sodipodi:linespacing="100%"
  4.1162           id="text3897"
  4.1163 @@ -2102,78 +2074,78 @@
  4.1164         xml:space="preserve"
  4.1165         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1166         x="755.43005"
  4.1167 -       y="-229.95343"
  4.1168 +       y="-449.95343"
  4.1169         id="text3909"
  4.1170         sodipodi:linespacing="100%"><tspan
  4.1171           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1172           sodipodi:role="line"
  4.1173           x="755.43005"
  4.1174 -         y="-229.95343"
  4.1175 +         y="-449.95343"
  4.1176           id="tspan3913">new work-unit has own</tspan><tspan
  4.1177           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1178           sodipodi:role="line"
  4.1179           x="755.43005"
  4.1180 -         y="-220.95343"
  4.1181 +         y="-440.95343"
  4.1182           id="tspan3927">local context, but</tspan><tspan
  4.1183           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1184           sodipodi:role="line"
  4.1185           x="756.78357"
  4.1186 -         y="-211.95343"
  4.1187 +         y="-431.95343"
  4.1188           id="tspan3931">prev VP suspended            </tspan></text>
  4.1189      <path
  4.1190         sodipodi:nodetypes="cc"
  4.1191         inkscape:connector-curvature="0"
  4.1192         id="path3953"
  4.1193 -       d="m 868.33356,-236.25901 40.86584,76.648"
  4.1194 +       d="m 868.33356,-456.25901 40.86584,76.648"
  4.1195         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1196      <text
  4.1197         xml:space="preserve"
  4.1198         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1199         x="357.27625"
  4.1200 -       y="-314.5929"
  4.1201 +       y="-534.5929"
  4.1202         id="text3955"
  4.1203         sodipodi:linespacing="100%"><tspan
  4.1204           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1205           sodipodi:role="line"
  4.1206           x="357.27625"
  4.1207 -         y="-314.5929"
  4.1208 +         y="-534.5929"
  4.1209           id="tspan3957">(in pthreads, checks</tspan><tspan
  4.1210           id="tspan3959"
  4.1211           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1212           sodipodi:role="line"
  4.1213           x="357.27625"
  4.1214 -         y="-305.5929"> mutex structures,</tspan><tspan
  4.1215 +         y="-525.5929"> mutex structures,</tspan><tspan
  4.1216           id="tspan3961"
  4.1217           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1218           sodipodi:role="line"
  4.1219           x="357.27625"
  4.1220 -         y="-296.5929"> cond var structs, etc)</tspan></text>
  4.1221 +         y="-516.5929"> cond var structs, etc)</tspan></text>
  4.1222      <text
  4.1223         xml:space="preserve"
  4.1224         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1225         x="459.43005"
  4.1226 -       y="-287.95343"
  4.1227 +       y="-507.95343"
  4.1228         id="text3963"
  4.1229         sodipodi:linespacing="100%"><tspan
  4.1230           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1231           sodipodi:role="line"
  4.1232           x="459.43005"
  4.1233 -         y="-287.95343"
  4.1234 +         y="-507.95343"
  4.1235           id="tspan3967">work-unit</tspan><tspan
  4.1236           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1237           sodipodi:role="line"
  4.1238           x="459.43005"
  4.1239 -         y="-278.95343"
  4.1240 +         y="-498.95343"
  4.1241           id="tspan3971">state chgs</tspan></text>
  4.1242      <text
  4.1243         sodipodi:linespacing="100%"
  4.1244         id="text3981"
  4.1245 -       y="24.04657"
  4.1246 +       y="-195.95343"
  4.1247         x="415.43005"
  4.1248         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1249         xml:space="preserve"
  4.1250         inkscape:transform-center-x="-8.125"><tspan
  4.1251 -         y="24.04657"
  4.1252 +         y="-195.95343"
  4.1253           x="415.43005"
  4.1254           sodipodi:role="line"
  4.1255           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1256 @@ -2182,39 +2154,39 @@
  4.1257         sodipodi:nodetypes="cc"
  4.1258         inkscape:connector-curvature="0"
  4.1259         id="path3386"
  4.1260 -       d="m 30.09217,-396.20795 134.75175,0"
  4.1261 +       d="m 30.09217,-616.20795 134.75175,0"
  4.1262         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#StopL);marker-end:url(#TriangleOutL)" />
  4.1263      <text
  4.1264         xml:space="preserve"
  4.1265         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1266         x="95.430077"
  4.1267 -       y="-397.95343"
  4.1268 +       y="-617.95343"
  4.1269         id="text3388"
  4.1270         sodipodi:linespacing="100%"><tspan
  4.1271           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
  4.1272           sodipodi:role="line"
  4.1273           x="95.430077"
  4.1274 -         y="-397.95343"
  4.1275 +         y="-617.95343"
  4.1276           id="tspan3390">Suspend at end of work-unit</tspan></text>
  4.1277      <text
  4.1278         xml:space="preserve"
  4.1279         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1280         x="29.276283"
  4.1281 -       y="-373.95343"
  4.1282 +       y="-593.95343"
  4.1283         id="text3392"
  4.1284         sodipodi:linespacing="100%"><tspan
  4.1285           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1286           sodipodi:role="line"
  4.1287           x="29.276283"
  4.1288 -         y="-373.95343"
  4.1289 +         y="-593.95343"
  4.1290           id="tspan3394">pthread is example of this case,</tspan><tspan
  4.1291           id="tspan3396"
  4.1292           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1293           sodipodi:role="line"
  4.1294           x="29.276283"
  4.1295 -         y="-364.95343">as is Cilk when suspends on sync</tspan></text>
  4.1296 +         y="-584.95343">as is Cilk when suspends on sync</tspan></text>
  4.1297      <g
  4.1298 -       transform="translate(297.05331,-684)"
  4.1299 +       transform="translate(297.05331,-904)"
  4.1300         id="g3398">
  4.1301        <text
  4.1302           xml:space="preserve"
  4.1303 @@ -2237,7 +2209,7 @@
  4.1304           style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
  4.1305      </g>
  4.1306      <g
  4.1307 -       transform="translate(177.05331,-684)"
  4.1308 +       transform="translate(177.05331,-904)"
  4.1309         id="g3406">
  4.1310        <text
  4.1311           xml:space="preserve"
  4.1312 @@ -2263,11 +2235,11 @@
  4.1313         sodipodi:nodetypes="cc"
  4.1314         inkscape:connector-curvature="0"
  4.1315         id="path3416"
  4.1316 -       d="m 428.09218,-396.20795 60.12116,0"
  4.1317 +       d="m 428.09218,-616.20795 60.12116,0"
  4.1318         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1319      <g
  4.1320         id="g3418"
  4.1321 -       transform="translate(-300,-60)">
  4.1322 +       transform="translate(-300,-280)">
  4.1323        <text
  4.1324           sodipodi:linespacing="100%"
  4.1325           id="text3420"
  4.1326 @@ -2299,27 +2271,27 @@
  4.1327         sodipodi:nodetypes="cc"
  4.1328         inkscape:connector-curvature="0"
  4.1329         id="path3428"
  4.1330 -       d="m 228.09216,-396.20795 21.72127,0"
  4.1331 +       d="m 228.09216,-616.20795 21.72127,0"
  4.1332         style="fill:none;stroke:#000000;stroke-width:0.99999994px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1333      <path
  4.1334         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1335 -       d="m 307.68401,-396.20795 61.46905,0"
  4.1336 +       d="m 307.68401,-616.20795 61.46905,0"
  4.1337         id="path3430"
  4.1338         inkscape:connector-curvature="0"
  4.1339         sodipodi:nodetypes="cc" />
  4.1340      <text
  4.1341         sodipodi:linespacing="100%"
  4.1342         id="text3432"
  4.1343 -       y="-363.95343"
  4.1344 +       y="-583.95343"
  4.1345         x="289.43005"
  4.1346         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1347         xml:space="preserve"><tspan
  4.1348           id="tspan3434"
  4.1349 -         y="-363.95343"
  4.1350 +         y="-583.95343"
  4.1351           x="289.43005"
  4.1352           sodipodi:role="line"
  4.1353           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">yes,</tspan><tspan
  4.1354 -         y="-354.95343"
  4.1355 +         y="-574.95343"
  4.1356           x="289.43005"
  4.1357           sodipodi:role="line"
  4.1358           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1359 @@ -2328,38 +2300,38 @@
  4.1360         xml:space="preserve"
  4.1361         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1362         x="315.43005"
  4.1363 -       y="-399.95343"
  4.1364 +       y="-619.95343"
  4.1365         id="text3438"
  4.1366         sodipodi:linespacing="100%"><tspan
  4.1367           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1368           sodipodi:role="line"
  4.1369           x="315.43005"
  4.1370 -         y="-399.95343"
  4.1371 +         y="-619.95343"
  4.1372           id="tspan3440">no</tspan></text>
  4.1373      <text
  4.1374         sodipodi:linespacing="100%"
  4.1375         id="text3442"
  4.1376 -       y="-373.95343"
  4.1377 +       y="-593.95343"
  4.1378         x="353.43005"
  4.1379         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1380         xml:space="preserve"><tspan
  4.1381           id="tspan3444"
  4.1382 -         y="-373.95343"
  4.1383 +         y="-593.95343"
  4.1384           x="353.43005"
  4.1385           sodipodi:role="line"
  4.1386           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">   done</tspan><tspan
  4.1387 -         y="-364.95343"
  4.1388 +         y="-584.95343"
  4.1389           x="353.43005"
  4.1390           sodipodi:role="line"
  4.1391           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1392           id="tspan3446">  with</tspan><tspan
  4.1393 -         y="-355.95343"
  4.1394 +         y="-575.95343"
  4.1395           x="353.43005"
  4.1396           sodipodi:role="line"
  4.1397           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1398           id="tspan3448">msgs</tspan></text>
  4.1399      <g
  4.1400 -       transform="translate(177.05331,-644)"
  4.1401 +       transform="translate(177.05331,-864)"
  4.1402         id="g3450">
  4.1403        <text
  4.1404           xml:space="preserve"
  4.1405 @@ -2395,11 +2367,11 @@
  4.1406         sodipodi:nodetypes="cc"
  4.1407         inkscape:connector-curvature="0"
  4.1408         id="path3460"
  4.1409 -       d="m 399.56974,-384.00562 0,12.75587"
  4.1410 +       d="m 399.56974,-604.00562 0,12.75587"
  4.1411         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1412      <g
  4.1413         id="g3472"
  4.1414 -       transform="translate(297.05331,-644)">
  4.1415 +       transform="translate(297.05331,-864)">
  4.1416        <text
  4.1417           sodipodi:linespacing="100%"
  4.1418           id="text3474"
  4.1419 @@ -2433,21 +2405,21 @@
  4.1420      <text
  4.1421         sodipodi:linespacing="100%"
  4.1422         id="text3486"
  4.1423 -       y="-434.5929"
  4.1424 +       y="-654.5929"
  4.1425         x="357.27625"
  4.1426         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1427         xml:space="preserve"><tspan
  4.1428           id="tspan3488"
  4.1429 -         y="-434.5929"
  4.1430 +         y="-654.5929"
  4.1431           x="357.27625"
  4.1432           sodipodi:role="line"
  4.1433           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1">(in pthreads, checks</tspan><tspan
  4.1434 -         y="-425.5929"
  4.1435 +         y="-645.5929"
  4.1436           x="357.27625"
  4.1437           sodipodi:role="line"
  4.1438           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1439           id="tspan3490"> mutex structures,</tspan><tspan
  4.1440 -         y="-416.5929"
  4.1441 +         y="-636.5929"
  4.1442           x="357.27625"
  4.1443           sodipodi:role="line"
  4.1444           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1445 @@ -2455,17 +2427,17 @@
  4.1446      <text
  4.1447         sodipodi:linespacing="100%"
  4.1448         id="text3494"
  4.1449 -       y="-407.95343"
  4.1450 +       y="-627.95343"
  4.1451         x="459.43005"
  4.1452         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1453         xml:space="preserve"><tspan
  4.1454           id="tspan3496"
  4.1455 -         y="-407.95343"
  4.1456 +         y="-627.95343"
  4.1457           x="459.43005"
  4.1458           sodipodi:role="line"
  4.1459           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">work-unit</tspan><tspan
  4.1460           id="tspan3498"
  4.1461 -         y="-398.95343"
  4.1462 +         y="-618.95343"
  4.1463           x="459.43005"
  4.1464           sodipodi:role="line"
  4.1465           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">state chgs</tspan></text>
  4.1466 @@ -2473,10 +2445,10 @@
  4.1467         sodipodi:nodetypes="cc"
  4.1468         inkscape:connector-curvature="0"
  4.1469         id="path3500"
  4.1470 -       d="m 548.67846,-395.848 261.45961,-0.0912"
  4.1471 +       d="m 548.67846,-615.848 261.45961,-0.0912"
  4.1472         style="fill:none;stroke:#000000;stroke-width:1.00000036px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1473      <g
  4.1474 -       transform="translate(717.05331,-684)"
  4.1475 +       transform="translate(717.05331,-904)"
  4.1476         id="g3518">
  4.1477        <text
  4.1478           xml:space="preserve"
  4.1479 @@ -2507,17 +2479,17 @@
  4.1480         sodipodi:nodetypes="cc"
  4.1481         inkscape:connector-curvature="0"
  4.1482         id="path3528"
  4.1483 -       d="m 868.0922,-396.20795 41.0608,0"
  4.1484 +       d="m 868.0922,-616.20795 41.0608,0"
  4.1485         style="fill:none;stroke:#000000;stroke-width:0.99999994px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1486      <path
  4.1487         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1488 -       d="m 519.56974,-384.00562 0,12.75587"
  4.1489 +       d="m 519.56974,-604.00562 0,12.75587"
  4.1490         id="path3568"
  4.1491         inkscape:connector-curvature="0"
  4.1492         sodipodi:nodetypes="cc" />
  4.1493      <g
  4.1494         id="g3572"
  4.1495 -       transform="translate(617.05331,-684)">
  4.1496 +       transform="translate(617.05331,-904)">
  4.1497        <text
  4.1498           sodipodi:linespacing="100%"
  4.1499           id="text3574"
  4.1500 @@ -2545,7 +2517,7 @@
  4.1501      </g>
  4.1502      <path
  4.1503         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1504 -       d="m 279.12967,-372.6255 -38.74926,53.74501 -0.37105,270.054168 35.3037,38.5653"
  4.1505 +       d="m 279.12967,-592.6255 -38.74926,53.74501 -0.37105,270.05417 35.3037,38.5653"
  4.1506         id="path3588"
  4.1507         inkscape:connector-curvature="0"
  4.1508         sodipodi:nodetypes="cccc" />
  4.1509 @@ -2553,11 +2525,11 @@
  4.1510         sodipodi:nodetypes="cc"
  4.1511         inkscape:connector-curvature="0"
  4.1512         id="path3590"
  4.1513 -       d="M 328.33352,22.451597 369.78149,-391.37328"
  4.1514 +       d="m 328.33352,-197.5484 41.44797,-413.82488"
  4.1515         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1516      <g
  4.1517         id="g4716"
  4.1518 -       transform="translate(-22.94669,-684)">
  4.1519 +       transform="translate(-22.94669,-904)">
  4.1520        <text
  4.1521           sodipodi:linespacing="100%"
  4.1522           id="text4718"
  4.1523 @@ -2587,65 +2559,65 @@
  4.1524         sodipodi:nodetypes="cc"
  4.1525         inkscape:connector-curvature="0"
  4.1526         id="path4726"
  4.1527 -       d="m 549.69914,-385.3604 79.38092,221.59022"
  4.1528 +       d="m 549.69914,-605.3604 79.38092,221.59022"
  4.1529         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1530      <text
  4.1531         xml:space="preserve"
  4.1532         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1533         x="573.43005"
  4.1534 -       y="-363.95343"
  4.1535 +       y="-583.95343"
  4.1536         id="text4728"
  4.1537         sodipodi:linespacing="100%"><tspan
  4.1538           id="tspan4732"
  4.1539           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1540           sodipodi:role="line"
  4.1541           x="573.43005"
  4.1542 -         y="-363.95343">No VPs</tspan><tspan
  4.1543 +         y="-583.95343">No VPs</tspan><tspan
  4.1544           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1545           sodipodi:role="line"
  4.1546           x="573.43005"
  4.1547 -         y="-354.95343"
  4.1548 +         y="-574.95343"
  4.1549           id="tspan4736">ready</tspan></text>
  4.1550      <text
  4.1551         inkscape:transform-center-x="-8.125"
  4.1552         xml:space="preserve"
  4.1553         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1554         x="451.43005"
  4.1555 -       y="-33.95343"
  4.1556 +       y="-253.95343"
  4.1557         id="text4791"
  4.1558         sodipodi:linespacing="100%"><tspan
  4.1559           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1560           sodipodi:role="line"
  4.1561           x="451.43005"
  4.1562 -         y="-33.95343"
  4.1563 +         y="-253.95343"
  4.1564           id="tspan4845">increase backoff each</tspan><tspan
  4.1565           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1566           sodipodi:role="line"
  4.1567           x="451.43005"
  4.1568 -         y="-24.95343"
  4.1569 +         y="-244.95343"
  4.1570           id="tspan8389">repetition &amp; update</tspan><tspan
  4.1571           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1572           sodipodi:role="line"
  4.1573           x="451.43005"
  4.1574 -         y="-15.95343"
  4.1575 +         y="-235.95343"
  4.1576           id="tspan8385">backoff state</tspan><tspan
  4.1577           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1578           sodipodi:role="line"
  4.1579           x="451.43005"
  4.1580 -         y="-6.9534302"
  4.1581 +         y="-226.95343"
  4.1582           id="tspan8462">Try to make core enter</tspan><tspan
  4.1583           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1584           sodipodi:role="line"
  4.1585           x="451.43005"
  4.1586 -         y="2.0465698"
  4.1587 +         y="-217.95343"
  4.1588           id="tspan8476">power-down idle state</tspan><tspan
  4.1589           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1590           sodipodi:role="line"
  4.1591           x="451.43005"
  4.1592 -         y="11.04657"
  4.1593 +         y="-208.95343"
  4.1594           id="tspan8464">while waiting</tspan></text>
  4.1595      <g
  4.1596 -       transform="translate(257.05331,-344)"
  4.1597 +       transform="translate(257.05331,-564)"
  4.1598         id="g4825">
  4.1599        <text
  4.1600           xml:space="preserve"
  4.1601 @@ -2676,22 +2648,22 @@
  4.1602         sodipodi:nodetypes="cc"
  4.1603         inkscape:connector-curvature="0"
  4.1604         id="path4839"
  4.1605 -       d="M 449.58066,-56.197783 313.82177,-9.623853"
  4.1606 +       d="m 449.58066,-276.19778 -135.75889,46.57393"
  4.1607         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1608      <path
  4.1609         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1610 -       d="m 328.29269,27.102941 c 201.947,-1.190476 188.59077,6.607148 222.99693,-14.360118 65.59054,-39.97116 58.47949,-168.794643 107.66658,-129.806543"
  4.1611 +       d="m 328.29269,-192.89706 c 201.947,-1.19048 188.59077,6.60715 222.99693,-14.36012 65.59054,-39.97116 58.47949,-168.79464 107.66658,-129.80654"
  4.1612         id="path7449"
  4.1613         inkscape:connector-curvature="0"
  4.1614         sodipodi:nodetypes="csc" />
  4.1615      <path
  4.1616         style="fill:none;stroke:#000000;stroke-width:0.99999994px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1617 -       d="m 659.56974,-13.52943 0,24.20748"
  4.1618 +       d="m 659.56974,-233.52943 0,24.20748"
  4.1619         id="path8339"
  4.1620         inkscape:connector-curvature="0"
  4.1621         sodipodi:nodetypes="cc" />
  4.1622      <g
  4.1623 -       transform="translate(80,300)"
  4.1624 +       transform="translate(80,80)"
  4.1625         id="g8347">
  4.1626        <text
  4.1627           xml:space="preserve"
  4.1628 @@ -2724,23 +2696,23 @@
  4.1629         xml:space="preserve"
  4.1630         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1631         x="681.43005"
  4.1632 -       y="-3.9534302"
  4.1633 +       y="-223.95343"
  4.1634         id="text8363"
  4.1635         sodipodi:linespacing="100%"><tspan
  4.1636           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1637           sodipodi:role="line"
  4.1638           x="681.43005"
  4.1639 -         y="-3.9534302"
  4.1640 +         y="-223.95343"
  4.1641           id="tspan8367">yes, send</tspan></text>
  4.1642      <path
  4.1643         sodipodi:nodetypes="cc"
  4.1644         inkscape:connector-curvature="0"
  4.1645         id="path8371"
  4.1646 -       d="M 630.69705,23.810904 509.08787,-52.944861"
  4.1647 +       d="M 630.69705,-196.1891 509.08787,-272.94486"
  4.1648         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1649      <g
  4.1650         id="g8391"
  4.1651 -       transform="translate(80,240)">
  4.1652 +       transform="translate(80,20)">
  4.1653        <text
  4.1654           sodipodi:linespacing="100%"
  4.1655           id="text8393"
  4.1656 @@ -2770,7 +2742,7 @@
  4.1657      </g>
  4.1658      <path
  4.1659         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:none;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1660 -       d="m 659.56974,-72.00562 0,13.287461"
  4.1661 +       d="m 659.56974,-292.00562 0,13.28746"
  4.1662         id="path8401"
  4.1663         inkscape:connector-curvature="0"
  4.1664         sodipodi:nodetypes="cc" />
  4.1665 @@ -2778,43 +2750,43 @@
  4.1666         xml:space="preserve"
  4.1667         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1668         x="625.43005"
  4.1669 -       y="-27.95343"
  4.1670 +       y="-247.95343"
  4.1671         id="text8403"
  4.1672         sodipodi:linespacing="100%"><tspan
  4.1673           id="tspan8405"
  4.1674           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1675           sodipodi:role="line"
  4.1676           x="625.43005"
  4.1677 -         y="-27.95343">Don't</tspan><tspan
  4.1678 +         y="-247.95343">Don't</tspan><tspan
  4.1679           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1680           sodipodi:role="line"
  4.1681           x="625.43005"
  4.1682 -         y="-18.95343"
  4.1683 +         y="-238.95343"
  4.1684           id="tspan8460">send</tspan></text>
  4.1685      <path
  4.1686         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1687 -       d="M 632.60182,-36.189098 509.09908,-56.55267"
  4.1688 +       d="M 632.60182,-256.1891 509.09908,-276.55267"
  4.1689         id="path8408"
  4.1690         inkscape:connector-curvature="0"
  4.1691         sodipodi:nodetypes="cc" />
  4.1692      <text
  4.1693         sodipodi:linespacing="100%"
  4.1694         id="text8420"
  4.1695 -       y="-97.95343"
  4.1696 +       y="-317.95343"
  4.1697         x="697.43005"
  4.1698         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1699         xml:space="preserve"><tspan
  4.1700           id="tspan8422"
  4.1701 -         y="-97.95343"
  4.1702 +         y="-317.95343"
  4.1703           x="697.43005"
  4.1704           sodipodi:role="line"
  4.1705           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">yes,</tspan><tspan
  4.1706 -         y="-88.95343"
  4.1707 +         y="-308.95343"
  4.1708           x="697.43005"
  4.1709           sodipodi:role="line"
  4.1710           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1711           id="tspan8424">got</tspan><tspan
  4.1712 -         y="-79.95343"
  4.1713 +         y="-299.95343"
  4.1714           x="697.43005"
  4.1715           sodipodi:role="line"
  4.1716           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1717 @@ -2822,41 +2794,41 @@
  4.1718      <text
  4.1719         sodipodi:linespacing="100%"
  4.1720         id="text8428"
  4.1721 -       y="-340.5929"
  4.1722 +       y="-560.5929"
  4.1723         x="737.27625"
  4.1724         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1725         xml:space="preserve"><tspan
  4.1726 -         y="-340.5929"
  4.1727 +         y="-560.5929"
  4.1728           x="737.27625"
  4.1729           sodipodi:role="line"
  4.1730           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1731           id="tspan8434">The difference between top two paths</tspan><tspan
  4.1732 -         y="-331.5929"
  4.1733 +         y="-551.5929"
  4.1734           x="737.27625"
  4.1735           sodipodi:role="line"
  4.1736           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1737           id="tspan8440">is the way the request hdlr+assigner has</tspan><tspan
  4.1738 -         y="-322.5929"
  4.1739 +         y="-542.5929"
  4.1740           x="737.27625"
  4.1741           sodipodi:role="line"
  4.1742           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1743           id="tspan8442">been implemented -- use shared state on</tspan><tspan
  4.1744 -         y="-313.5929"
  4.1745 +         y="-533.5929"
  4.1746           x="737.27625"
  4.1747           sodipodi:role="line"
  4.1748           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1749           id="tspan8450">top path, but only local on the second.</tspan><tspan
  4.1750 -         y="-304.5929"
  4.1751 +         y="-524.5929"
  4.1752           x="737.27625"
  4.1753           sodipodi:role="line"
  4.1754           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1755           id="tspan8454">NOTE: the request hdlr and assigner are</tspan><tspan
  4.1756 -         y="-295.5929"
  4.1757 +         y="-515.5929"
  4.1758           x="737.27625"
  4.1759           sodipodi:role="line"
  4.1760           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1761           id="tspan8456">combined into a single straight-line piece</tspan><tspan
  4.1762 -         y="-286.5929"
  4.1763 +         y="-506.5929"
  4.1764           x="737.27625"
  4.1765           sodipodi:role="line"
  4.1766           style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1767 @@ -2865,28 +2837,28 @@
  4.1768         sodipodi:nodetypes="cccc"
  4.1769         inkscape:connector-curvature="0"
  4.1770         id="path8466"
  4.1771 -       d="m 279.12967,-254.53026 -22.71235,36.22941 -0.21748,176.328743 29.26421,32.663466"
  4.1772 +       d="m 279.12967,-474.53026 -22.71235,36.22941 -0.21748,176.32874 29.26421,32.66347"
  4.1773         style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1774      <path
  4.1775         sodipodi:nodetypes="cc"
  4.1776         inkscape:connector-curvature="0"
  4.1777         id="path8468"
  4.1778 -       d="M 548.67838,-278.64159 808.48533,-388.65596"
  4.1779 +       d="M 548.67838,-498.64159 808.48533,-608.65596"
  4.1780         style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)" />
  4.1781      <text
  4.1782         sodipodi:linespacing="100%"
  4.1783         id="text8470"
  4.1784 -       y="-263.95343"
  4.1785 +       y="-483.95343"
  4.1786         x="571.43005"
  4.1787         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1788         xml:space="preserve"><tspan
  4.1789 -         y="-263.95343"
  4.1790 +         y="-483.95343"
  4.1791           x="571.43005"
  4.1792           sodipodi:role="line"
  4.1793           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1794           id="tspan8472">No VPs</tspan><tspan
  4.1795           id="tspan8474"
  4.1796 -         y="-254.95343"
  4.1797 +         y="-474.95343"
  4.1798           x="571.43005"
  4.1799           sodipodi:role="line"
  4.1800           style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1">  ready</tspan></text>
  4.1801 @@ -2906,7 +2878,7 @@
  4.1802      <flowRoot
  4.1803         xml:space="preserve"
  4.1804         id="flowRoot8540"
  4.1805 -       style="fill:black;stroke:none;stroke-opacity:1;stroke-width:1px;stroke-linejoin:miter;stroke-linecap:butt;fill-opacity:1;font-family:Sans;font-style:normal;font-weight:normal;font-size:8px;line-height:125%;letter-spacing:0px;word-spacing:0px"><flowRegion
  4.1806 +       style="font-size:8px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans"><flowRegion
  4.1807           id="flowRegion8542"><rect
  4.1808             id="rect8544"
  4.1809             width="527.61902"
  4.1810 @@ -2914,11 +2886,11 @@
  4.1811             x="457.14285"
  4.1812             y="491.66666"
  4.1813             style="font-size:8px" /></flowRegion><flowPara
  4.1814 -         id="flowPara8546"></flowPara></flowRoot>    <flowRoot
  4.1815 +         id="flowPara8546" /></flowRoot>    <flowRoot
  4.1816         xml:space="preserve"
  4.1817         id="flowRoot8548"
  4.1818         style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans"
  4.1819 -       transform="translate(3.5307846,-441.70658)"><flowRegion
  4.1820 +       transform="translate(-336.46922,-441.70658)"><flowRegion
  4.1821           id="flowRegion8550"><rect
  4.1822             id="rect8552"
  4.1823             width="539.04761"
  4.1824 @@ -2953,14 +2925,166 @@
  4.1825           id="flowPara8801">Core-controller was a second level beneath the runtime (Language Animator is the runtime)..  The sched slots were virtual physical animators, and the masterVP was the Language Animator, which the AnimationMaster Fn and the plugin Fns supplied the behavior of.  The coreCtlr switched between levels -- The MasterVP was &quot;outside&quot; the framework of the schedling slots -- in a way it was beside them (they took turns getting the actual phys animator, so same level) and in other way was above -- it controlled what went into the slots, so above them.  I another way, the MasterVP was the Language Animator, while the slots were virtual physical, so they had no locial connection -- the slots were related to the same thing that animatored the language animator..  a very strange arrangement.</flowPara></flowRoot>    <text
  4.1826         xml:space="preserve"
  4.1827         style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans"
  4.1828 -       x="890.4762"
  4.1829 -       y="479.28571"
  4.1830 +       x="894.00696"
  4.1831 +       y="-182.42088"
  4.1832         id="text8833"
  4.1833 -       sodipodi:linespacing="125%"
  4.1834 -       transform="translate(3.5307846,-441.70658)"><tspan
  4.1835 +       sodipodi:linespacing="125%"><tspan
  4.1836           sodipodi:role="line"
  4.1837           id="tspan8835"
  4.1838 -         x="890.4762"
  4.1839 -         y="479.28571"> </tspan></text>
  4.1840 +         x="894.00696"
  4.1841 +         y="-182.42088"> </tspan></text>
  4.1842 +    <g
  4.1843 +       id="g3490"
  4.1844 +       transform="translate(177.05331,-343.31973)">
  4.1845 +      <text
  4.1846 +         sodipodi:linespacing="100%"
  4.1847 +         id="text3492"
  4.1848 +         y="285.39841"
  4.1849 +         x="222.37003"
  4.1850 +         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1851 +         xml:space="preserve"><tspan
  4.1852 +           y="285.39841"
  4.1853 +           x="222.37003"
  4.1854 +           sodipodi:role="line"
  4.1855 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
  4.1856 +           id="tspan3501">send work-</tspan><tspan
  4.1857 +           y="294.39841"
  4.1858 +           x="222.37003"
  4.1859 +           sodipodi:role="line"
  4.1860 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
  4.1861 +           id="tspan3511">push msg</tspan><tspan
  4.1862 +           y="303.39841"
  4.1863 +           x="223.72354"
  4.1864 +           sodipodi:role="line"
  4.1865 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
  4.1866 +           id="tspan3513">and start </tspan><tspan
  4.1867 +           y="312.39841"
  4.1868 +           x="222.37003"
  4.1869 +           sodipodi:role="line"
  4.1870 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
  4.1871 +           id="tspan3515">timer for it.</tspan></text>
  4.1872 +      <rect
  4.1873 +         style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0"
  4.1874 +         id="rect3503"
  4.1875 +         width="57.486332"
  4.1876 +         height="63.639675"
  4.1877 +         x="193.78287"
  4.1878 +         y="274.77512" />
  4.1879 +    </g>
  4.1880 +    <text
  4.1881 +       xml:space="preserve"
  4.1882 +       style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans"
  4.1883 +       x="84.168976"
  4.1884 +       y="-681.48474"
  4.1885 +       id="text3537"
  4.1886 +       sodipodi:linespacing="125%"><tspan
  4.1887 +         sodipodi:role="line"
  4.1888 +         id="tspan3539"
  4.1889 +         x="84.168976"
  4.1890 +         y="-681.48474"
  4.1891 +         style="font-size:40px">Control Flow of Universal Proto-Runtime </tspan></text>
  4.1892 +    <g
  4.1893 +       transform="translate(237.05331,-404)"
  4.1894 +       id="g3541">
  4.1895 +      <text
  4.1896 +         xml:space="preserve"
  4.1897 +         style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1898 +         x="222.25137"
  4.1899 +         y="285.36765"
  4.1900 +         id="text3543"
  4.1901 +         sodipodi:linespacing="100%"><tspan
  4.1902 +           id="tspan3547"
  4.1903 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
  4.1904 +           sodipodi:role="line"
  4.1905 +           x="222.25137"
  4.1906 +           y="285.36765">Expired timer</tspan><tspan
  4.1907 +           style="font-size:9px;text-align:center;text-anchor:middle;fill:#900600;fill-opacity:1"
  4.1908 +           sodipodi:role="line"
  4.1909 +           x="222.25137"
  4.1910 +           y="294.36765"
  4.1911 +           id="tspan3553">handler</tspan></text>
  4.1912 +      <rect
  4.1913 +         y="274.77512"
  4.1914 +         x="193.78287"
  4.1915 +         height="25.544432"
  4.1916 +         width="57.486332"
  4.1917 +         id="rect3549"
  4.1918 +         style="fill:none;stroke:#000000;stroke-width:1.08018124;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" />
  4.1919 +    </g>
  4.1920 +    <path
  4.1921 +       style="fill:none;stroke:#000000;stroke-width:1.00000012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-8-7-0)"
  4.1922 +       d="m 323.63077,-165.12086 131.74691,33.98134"
  4.1923 +       id="path3555"
  4.1924 +       inkscape:connector-curvature="0"
  4.1925 +       sodipodi:nodetypes="cc" />
  4.1926 +    <text
  4.1927 +       xml:space="preserve"
  4.1928 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1929 +       x="393.43005"
  4.1930 +       y="-151.95343"
  4.1931 +       id="text3557"
  4.1932 +       sodipodi:linespacing="100%"><tspan
  4.1933 +         style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1934 +         sodipodi:role="line"
  4.1935 +         x="393.43005"
  4.1936 +         y="-151.95343"
  4.1937 +         id="tspan3561">  Expired</tspan><tspan
  4.1938 +         style="font-size:9px;text-align:center;text-anchor:middle;fill:#008000;fill-opacity:1"
  4.1939 +         sodipodi:role="line"
  4.1940 +         x="393.43005"
  4.1941 +         y="-142.95343"
  4.1942 +         id="tspan3565">                  timer</tspan></text>
  4.1943 +    <text
  4.1944 +       xml:space="preserve"
  4.1945 +       style="font-size:10px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;writing-mode:lr-tb;text-anchor:start;fill:#d45500;fill-opacity:1;stroke:none;font-family:Trebuchet MS;-inkscape-font-specification:Trebuchet MS"
  4.1946 +       x="449.27625"
  4.1947 +       y="-93.273193"
  4.1948 +       id="text3567"
  4.1949 +       sodipodi:linespacing="100%"><tspan
  4.1950 +         id="tspan3579"
  4.1951 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1952 +         sodipodi:role="line"
  4.1953 +         x="449.27625"
  4.1954 +         y="-93.273193">When send work to other core,</tspan><tspan
  4.1955 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1956 +         sodipodi:role="line"
  4.1957 +         x="449.27625"
  4.1958 +         y="-84.273193"
  4.1959 +         id="tspan3591">it may be sleeping. So when timer</tspan><tspan
  4.1960 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1961 +         sodipodi:role="line"
  4.1962 +         x="449.27625"
  4.1963 +         y="-75.273193"
  4.1964 +         id="tspan3593">expires, do escrow, which is atomic</tspan><tspan
  4.1965 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1966 +         sodipodi:role="line"
  4.1967 +         x="449.27625"
  4.1968 +         y="-66.273193"
  4.1969 +         id="tspan3595">check of whether work was received.</tspan><tspan
  4.1970 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1971 +         sodipodi:role="line"
  4.1972 +         x="449.27625"
  4.1973 +         y="-57.273193"
  4.1974 +         id="tspan3597">If not, then check if dest core is </tspan><tspan
  4.1975 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1976 +         sodipodi:role="line"
  4.1977 +         x="449.27625"
  4.1978 +         y="-48.273193"
  4.1979 +         id="tspan3599">sleeping -- if no, then take work</tspan><tspan
  4.1980 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1981 +         sodipodi:role="line"
  4.1982 +         x="449.27625"
  4.1983 +         y="-39.273193"
  4.1984 +         id="tspan3611">back. If yes, do wakeup and set a </tspan><tspan
  4.1985 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1986 +         sodipodi:role="line"
  4.1987 +         x="449.27625"
  4.1988 +         y="-30.273193"
  4.1989 +         id="tspan3609">new timer, which will do  this</tspan><tspan
  4.1990 +         style="font-size:9px;text-align:start;text-anchor:start;fill:#d45500;fill-opacity:1"
  4.1991 +         sodipodi:role="line"
  4.1992 +         x="449.27625"
  4.1993 +         y="-21.273193"
  4.1994 +         id="tspan3617">handling again.</tspan></text>
  4.1995    </g>
  4.1996  </svg>

     5.1 --- a/0__Papers/VMS/Universal/latex/VMS_universal.tex	Fri May 04 09:44:14 2012 -0700
     5.2 +++ b/0__Papers/VMS/Universal/latex/VMS_universal.tex	Wed May 16 06:32:02 2012 -0700
     5.3 @@ -33,7 +33,7 @@
     5.4  
     5.5  %+Abstract
     5.6  \begin{abstract}
     5.7 -Software has not been keeping up with parallel hardware, which slows the economy and retards adoption of new hardware. The gap is due in part to the disruption caused by  moving to parallel languages, and in part to the prohibitive effort of porting application code across platforms.  A leading idea for solving this is domain-specific parallel languages, where custom constructs are made to match features of the problem.  However, such languages have a small number of users, which can't support the currently large effort to create such languages and port them across hardware platforms.
     5.8 +Software has not been keeping up with parallel hardware, which slows the economy and retards adoption of new hardware. The gap is due in part to the disruption caused by  moving to parallel languages, and in part to the prohibitive effort of porting parallel code across platforms.  A leading idea for solving this is domain-specific parallel languages, where custom constructs are made to match features of the problem.  However, such languages have a small number of users, which can't support the currently large effort to create such languages and port them across hardware platforms.
     5.9  
    5.10  To simplify creation of domain-specific languages, we propose a "proto" runtime, which provides the cross-language portion of runtime behavior. This includes handling the concurrency issues within the runtime itself.  A given language only provides sequential implementations of its constructs.
    5.11  
    5.12 @@ -45,30 +45,30 @@
    5.13  
    5.14  
    5.15  \section{Introduction}
    5.16 -Current parallel languages, such as pthreads, TBB, OpenMP, and MPI, require programmers to learn new, complex mental models. Sequential programmers have to be retrained to start using them, and a new set of programming practices must be adopted for them. Such retraining generates a large disruption in application-software companies that attempt to start using these parallel languages.
    5.17 +Current parallel languages, such as pthreads, TBB, OpenMP, and MPI, require programmers to learn new, complex mental models. Sequential programmers have to be retrained to  using the parallel languages, and a new set of programming practices must be adopted for them. Such retraining generates a large disruption in the application-software companies that attempt to start using these parallel languages.
    5.18  
    5.19 -Further, after learning the new language and adopting the new programming practices, the companies still have to hand-tune each application for each target hardware platform. It also means that customers have to get a new binary when they upgrade hardware. Both effects are costly, and tend to retard adoption of new hardware, despite potential performance gains.
    5.20 +Further, after learning the new language and adopting the new programming practices, the companies still have to hand-tune each application for each target hardware platform. Not only that, but customers have to acquire a new binary when they upgrade hardware. Both effects are costly, and tend to retard adoption of new hardware, despite potential performance gains.
    5.21   
    5.22 -The net effect is that software lags behind hardware, and the potential advantages across the economy of new hardware designs are lost. What is needed is a way to ease the transition from sequential programming to parallel programming, and to reduce the need for hand-tuning to get efficient performance on new hardware.
    5.23 +The net result is that software lags behind hardware, and the potential advantages across the economy of new hardware designs are lost. What is needed is a way to ease the transition from sequential programming to parallel programming, and to reduce the amount of hand-tuning done to get efficient performance on new hardware.
    5.24  
    5.25   
    5.26 -Domain-specific languages promise to deliver both easier transition and efficiency across hardware[]. They do this by providing custom constructs that match patterns in the application. Thus, they are natural for the sequential programmer to use. The constructs ``hide'' the parallelism inside themselves, letting the runtime or toolchain handle it, freeing the programmer. 
    5.27 +Domain-specific languages promise to deliver both easier transition and efficiency across hardware[]. They do this by providing custom constructs that match to patterns in the application. Thus, they are natural for the sequential programmer to use. The custom constructs ``hide''  parallelism inside themselves, letting the runtime or toolchain handle it, freeing the programmer, and putting performance inside the tools. This makes coding easier and reduces hand-tuning.
    5.28  
    5.29  To illustrate the concept, we briefly cover one such language, which is used for describing the behavior of hardware, called HWSim. It has a straight-forward means for describing hardware, and then extracts parallelism from the description. The extracted parallelism is then efficiently exploited  on a variety of target platforms.
    5.30  
    5.31 -One blockage to wide-spread adoption of such domain-specific languages is the cost of developing them. Currently, toolchains are typically created, with an optimizer and back end for each target platform, which is expensive to create.
    5.32 +One blockage to wide-spread adoption of such domain-specific languages is the cost of developing them. Currently, a toolchain is created, including an optimizer and back end for each target platform. This is expensive.
    5.33  
    5.34  An alternative approach is so-called embedded languages, in which the new constructs are viewed as library calls made from a base language, such as C. This relies on the runtime system to provide efficient execution on a given target platform. It avoids the expense of creating a toolchain for each target, by instead creating a runtime for each target that is tuned to the hardware.
    5.35  
    5.36  This is a step forward, but such runtimes are still expensive to create. A technology to simplify the runtime creation would be helpful.
    5.37  
    5.38 -In this paper, we present a means to reduce runtime creation, by not only reducing the complexity, but also by making reuse across languages more practical. It essentially breaks a runtime into two pieces: a part that implements the semantic behavior of the custom constructs, and a part that is the same for every language.
    5.39 +In this paper, we present a means to reduce runtime creation, by not only reducing the complexity, but also by making reuse of runtime code in new languages more practical. It essentially breaks a runtime into two pieces: a part that implements the semantic behavior of the custom constructs, and a part that is the same for every language.
    5.40  
    5.41 - A nice happenstance is that the complicated multi-threaded issues that come up inside current runtime implementations can be collected inside the part that stays the same across languages. This means the language only need supply a sequential implementation of its constructs' behavior.
    5.42 + A nice happenstance is that most of the complicated concurrency issues that come up inside current runtime implementations can be collected inside the part that stays the same across languages. This means the language only need supply a sequential implementation of its constructs' behavior. The cross-language part handles low-level synchronization and communication between cores, exporting a simple interface. 
    5.43  
    5.44  Another nice happenstance is that the interface between the two parts of the runtime modularizes the runtime code. This, by itself, speeds development of new runtimes. In addition, it makes sharing between languages practical, especially for the portion that chooses where to perform work, and in which order.
    5.45  
    5.46 - This portion of the runtime handles data affinity and the shape of the dependency graph, which are responsible for the resulting performance. However, it contains few, if any, language-specific portions, so is practical to share between languages, for a given target platform.  This saves a non-trivial amount of development work.
    5.47 + This placement-choosing portion handles data affinity and the shape of the dependency graph, which are largely responsible for the resulting performance. However, it can be written to contain few language-specific portions, making it practical to share between languages, for a given target platform.  This saves a non-trivial amount of development work.
    5.48  
    5.49  While runtimes built without our contribution are still free to share such code, they have no equivalent interface between the runtime pieces. This makes isolating this portion of code more time consuming, and forces more effort to fit code from one runtime into the one of a different language.
    5.50  
    5.51 @@ -79,9 +79,17 @@
    5.52  Organization of paper
    5.53  
    5.54  \section{Background and Related Work}
    5.55 -For performance, the proto-runtime supports multiple levels of runtime hierarchy. In higher levels, work-units are large, leaving time for the decision about where to execute them to use advanced algorithms, which track data affinity and analyze dependency patterns. For lower levels, the work-units are smaller, leaving less time to search for the best location, so they have simpler algorithms.
    5.56  
    5.57 -UPR differs from pthreads, TBB, and other thread packages in that it provides more services to simplify runtime creation, and, more importantly, UPR has a mental model that is specific to runtime creation.  pthreads and TBB are programming languages in their own right -- but UPR has no semantics usable in application code, because it's only a  \emph{part} of a runtime. 
    5.58 +The current method of creating a runtime for a new domain-specific language is either writing one from scratch, normally on top of pthreads, or else modifying an existing runtime. LibGomp, the runtime for OpenMP, is a popular starting point.
    5.59 +
    5.60 +From anecdotal evidence, talking with 5 groups that have created domain specific languages, development of bare functionality of the runtime takes anywhere from one to several months. Perhaps a majority  of the time is spent  on multi-threading design and debugging.  That is just for basic functionality; if they want low overhead, they expect another three to six months of performance tuning.
    5.61 +
    5.62 +One group was interviewed, which used libGomp as the base upon which to build a streaming dataflow language.
    5.63 +
    5.64 +UPR is designed to shorten that development time.
    5.65 +
    5.66 +
    5.67 +UPR differs from pthreads, TBB, and other thread packages in that it provides a mental model that is specific to runtime creation.  Pthreads and TBB are programming languages in their own right, but UPR is not. It has no semantics usable in application code, because it's only a  \emph{part} of a runtime. It is geared exclusively to runtime creation. 
    5.68  
    5.69  ?
    5.70