[Transformers] The ASF+SDF Meta-Environment 2.0-RC1 (Release Candidate)

De : "Jurgen J. Vinju" &lt;Jurgen.Vinju(a)cwi.nl&gt; Date : 16 mai 2006 10:31:14 HAEC À : meta-users-list(a)cwi.nl Objet : [meta-users-list] The ASF+SDF Meta-Environment 2.0-RC1 (Release Candidate) Répondre à : meta-users-list(a)cwi.nl Dear ASF+SDF Meta-Environment people, We have decided to put a Release Candidate of the full ASF+SDF Meta-Environment online: version 2.0-RC1 It has been a while since the previous release, and a lot of work has been done: - many requests for enhancement and bugfixes have been implemented. - a number of experimental features have been implemented - some features have been optimized for speed or memory consumption - and some legacy implementations have been replaced This is a Release Candidate, because we think a lot more has to be done to finish version 2.0 as a product: - some features are only half implemented (e.g. only by the ASF interpreter, not by the ASF compiler) - some features are not optimized (too slow or too big) - the documentation is out of date Below you will find the only available documentation on what changed and what you need to do to get your specifications working. We ARE catching up on the documentation front. ================= BASIC INFORMATION ================= Website ------- * http://www.meta-environment.org Help ---- * Problems, bugs and enhancements: * mailto:meta-support- list(a)cwi.nl * General help (also by other users): * mailto:meta-users- list(a)cwi.nl * Submitting issues directly: * bugzilla.sen.cwi.nl:8080/ index.cgi Required software ----------------- * The following software is necessary to compile and run 2.0RC1: * GNU gcc 3.4.x (the 4.x series contains bugs that we can not work around) * GNU make * Graphviz dot >= 1.12 * Java JRE 1.5 (will not work with 1.4.x) ============== CHANGES & TIPS ============== Commandline ----------- * The ASF+SDF Meta-Environment starts up with the command "asfsdf-meta", not with "meta" anymore. * The SDF Meta-Environment (without ASF) can be started with: "sdf-meta". * The dump scripts, pt-dump, eqs-dump, def-dump, etc. do not have options to save the "term-store" anymore. This feature has dissappeared and we are thinking about better ways to cache intermediate results. User interface -------------- * The third-party editor interface is gone. So no more emacs or vi. * The editors are now implemented using a standard Swing implementation. * All Meta-Environment GUI tools are implemented in Java/Swing and hosted in a dockable window environment, implemented by InfoNode.com (GPL) * Note that many menu options have keyboard shortcuts, and all menus have changed. * Actions->Parse does not exist anymore, after saving an editor (CTRL+S), all necessary updates are done to the state of the system automatically. Any errors that may be the result of your changes are listed in the tab of the error viewer. (This is one of the experimental features that are not fast enough yet, sometimes after a save it takes a long time for the system to become idle again) * The full state of the system is viewable from the 'Progress' viewer tab. * All editors apply a generic syntax highlighting scheme, which you can influence in a number of ways. Contact meta-support- list(a)cwi.nl for questions. * We use "prefuse", an open source graph visualization and animation package: www.prefuse.org. The graphviz layout algorithm is used, but the actual displaying is done by prefuse: * left click and drag: move a node * right click and drag up/down: zoom out/in * right click on canvas: zoom to fit * left click on node: select node * right click on node: open context popup menu * the "Graph" menu contains some other options * Switch workspace: The "Workspace" is the directory where new modules will appear and existing modules are found. The File menu contains a switch workspace button that can be used at anytime. * Open module: the file browser that is opened gives access to: * the Workspace * the SDF library * the ASF+SDF library You should not open files outside of these domains, the interface tries to limit this, but it is not hacker proof. * Note that syntax definitions are now in the SDF library, while ASF+SDF data-structures and utilities are still in the ASF+SDF library. * Library modules are read-only, and coloured grey in the import graph. * The save termstore feature has dissappeared. We are thinking about a replacement for caching intermediate results between runs of the Meta-Environment. Syntax Definition Formalism and SGLR ------------------------------------ * The SDF library now comes with some new grammars: * C grammar * Parameterized C preprocessor grammar * The syntax of SDF has been redesigned to use the same syntax for string literals everywhere. This also implies that unquoted string literal symbols are not literals anymore. You will get parse errors in your SDF modules. Before, the SDF checker would warn you about deprecated use of unquoted literals. * The AsFix2ME format has changed. It does not flatten lexical syntax anymore. Fully structured lexical trees are generated from now on. * AsFix2ME also does some subtle rearrangements of LAYOUT nodes. For details read the source code or contact meta-support- list(a)cwi.nl * The SDF checker generates less false warnings on old issues, and some more false warnings for new issues. * CASE-INSENSITIVE LITERALS are added as a feature to SDF2. The syntax is as follows: 'begin' STAT* 'end' -> PROGRAM Defines a program with two case-insensitive literals begin and end. (use of single quotes instead of double quotes). The SDF normalizer generates for you: [bB][eE][gG][iI][nN] -> 'begin' [eE][nN][dD] -> 'end' remember the case-sensitive notation is still: "begin" STAT* "end" -> PROGRAM which generates: [b][e][g][i][n] -> "begin" [e][n][d] -> "end" * ci-lit(<str>) was added as a symbol constructor to AsFix2[ME] * CYCLIC GRAMMARS: SGLR now accepts cyclic grammars. A parse forest with a representation of a cycle will be generated instead of a parse error. The representation is NOT minimal. Contact meta-support-list(a)cwi.nl for details. * AsFix2[ME] is extended with a new Tree constructor, next to appl, amb and char we now have: cycle. * The backend of SGLR was optimized for highly ambiguous languages. The use of memoization in several algorithms has improved the worst-case complexity from exponential to polynominal. * SGLR does not provide a detailed ambiguity report anymore, use the renewed implementation of the tool "ambtracker". * NOTE: we are currently reimplementing the API's around SGLR and remodularizing its implementation. * Syntax productions are generated from priority sections like was done several versions back. BEWARE: we removed this then because the attributes of productions are merged if the rest of the production is the same. It might lead to unexpected shapes of productions in an AsFix2 parse forest. The benefit in grammars for C, Java and C++ is so big, that we decided to put it back in (several dozens of lines less in SDF modules with large expression grammars) * PRIORITY FOR SELECTED ARGUMENTS. A priority can now be limited to be applicable in a certain argument only. This works for single productions as well as groups, and priorities are still transitively closed. Example: context-free priorities { E "[" E "]" -> E E "bla" -> E } <0> > E "*" E -> E {left} The first group has a higher priority than the second group, but only in the first (0) argument. Between the <..> brackets comma separated lists of argument numbers are accepted. We start counting at zero, and count EVERY MEMBER of the left hand side including literals, counting nested symbols as 1 member, and ignoring the implicit LAYOUT? non-terminal. Algebraic Specification Formalism --------------------------------- * STRUCTURED LEXICALS: this is the new feature that breaks old ASF+SDF programs that made use of lexical constructor functions. This is a long story, here is the summary: * The CHAR sort is not to be used by you anymore. From now on you use 'lexical variables' as such defined in SDF2: lexical syntax [0-9]+ -> Number Number "." Number -> Real lexical variables "digit"[0-9]* -> [0-9] "digits"[0-9]* -> [0-9]+ "num"[0-9]* -> Number "real"[0-9]* -> Real * In other words, for lexical non-terminals and character classes do not define normal variables, but lexical variables. * Variables that range over SUBCLASSES may be used. If a variable does not range over a subclass of the expected character class, an error will be generated. Example: lexical variables "half" -> [0-4] "more" -> [0-9A-F] The variable 'half' is acceptable everywhere [0-9] is also acceptible. The variable 'more' is not accepted at [0-9] locations. * Lexical constructors are now fully NESTED, because lexical syntax is not flattened to a list representation anymore, we now also need STRUCTURED LEXICAL CONSTRUCTORS. * For each production in lexical syntax a lexical constructor function is generated: - the prefix function name is the sort name in lowercase - each function has as many arguments as the production had members, including literals. - literals are to be typed literally - characters are escaped in the same manner as characters are escaped in the definitions of SDF2 character classes. - lexical variables are only accepted in the context of these generated constructor functions - example (see syntax and variables above): equations [zeros] number(0 digits) = number (digits) [trunk] real(num . number(digits2)) = real (num . 0) - another example: lexical syntax [\"] ~[\"]* [\"] -> String lexical variables "char"[0-9]* -> ~[\"] "chars"[0-9]* -> ~[\"]* equations [remove-nl] string(\" chars1 \n chars2 \") = string(\" chars1 chars2 \") - You can not construct any lexical that does not conform to the SDF definition of the lexical syntax anymore. Not even as an intermediate step. This sometimes requires some extra thinking. See the library modules basic/Identifiers and basic/Strings and basic/Bytes for examples on how to analyse or transform lexical syntax. * (Traversal) Functions that returns lists, there used to be many problems with functions that returns lists, including traversal functions that visit list nodes: e.g. f(A) -> {B ","}* SDF, AsFix2 and ASF have been changed to fix these issues. * REWRITING LAYOUT. Layout nodes are now first class citizens in ASF specifications. Each layout node in a syntactic pattern (i.e. left-hand sides or right-hand sides of conditions and equations) can now be in either one of these two modes: * IGNORED: the layout is ignored by a match, and put in literally by a constructing pattern. The syntax for this is just as it used to be: simply type the layout as it is defined and it will be ignored. * NOT IGNORED: the layout is matched and constructed according to your specification. The mode is activated when: * You use a lexical constructor function for layout * You use 'layout brackets' * Examples: lexical syntax [\ \t\n] -> LAYOUT "%%" ~[\n]* [\n] -> LAYOUT context-free syntax "_" "_" -> DUMMY {layout-bracket} lexical variables "L"[0-9]* -> LAYOUT equations [] if !E then S*1 else S*2 fi = if E then S*2 else S*1 fi [] if layout(\n) E then S* fi = if layout(\ ) E then S* fi [] if L1 layout(\n) L2 E then S* fi = if L1 layout(\ ) L2 E then S* fi [] _ layout(\t) _ = _ layout(\ ) layout(\ ) _ * The first equation ignores layout completely, like usual. * The second equation only matches an if statement if there is EXACTLY one newline between the if and the E. * The third equation matches an if statement with a newline SOMEWHERE between the if and the E. * The fourth equation matches ALL single tabs and translates them to two spaces. * Condition syntax * The deprecated '=' condition operator has dissappeared * Integrated unit tests * There are still unit tests in ASF specifications, example: equations [] myfunction(...) = myanswer(...) tests [] myfunction(xyz) == myanswer(abc) * Rewriting with pos-info annotations and others * You can store and retrieve parse tree annotations using the modules utilities/Annotations, utilities/ATermAnnotations and utilities/PosInfo. * The ASF interpreter supports this by default * The ASF compiler has an extra commandline option to activate support for annotations in the generated code. * New builtin functions: * See utilities/Parsing for calling the parser on files and strings * See utilities/Files for reading and writing files from ASF * The commandline versions of the interpreter and the compiler have a new option to pass a parse table in, for use with the parsing builtin * Interface of compiled specifications * A compiled specification can now read source code directly, without using sglr first. * If you compiled a specification from the Meta-Environment directly, this is the default behavior: the binary will rewrite source- to-source. * Wild and strict variables (not experimental anymore) * The experimental feature of wild and strict variables is consolidated. * example: variables "_x"[0-9]* -> Integer {wild} "x"[0-9]* -> Integer {strict} * wild variables may only be used as 'wildcards'. No further use of a variable is allowed after using it in a match. * strict variables are used for information propagation. A strict variable MUST be used after it's introduction. * NOTE: the use of strict variables leads to less errors, it prohibits unintended loss of information. * NOTE: the use of wild variables also leads to less errors, it prohibits unintended failure of matches by use of the same variable names twice. * Complete functions * An experimental feature: complete functions. You can annotate a function to be complete, which means that it may not appear in a normal form. Use this if you intend to implement a function that deals with all input correctly. * You will get a run-time error (interpreter only) if a complete function does not reduce. * Example: context-free syntax myfunction(Statement) -> Booleans {complete} * Rewriting ambiguities * The rewriting of ambiguities implementation has changed recently, but not tested thoroughly yet. Use with caution.