pydsl
PyDSL is a grammar library written in Python. It includes parsers, validation tools and encoding tools.
The source code is hosted in github and the binary releases in pypi.
The basics
- Grammars: a set of rules that defines a language and can expand the meaning of an input
- Verbs: actions performed with grammars and input data
- Library
- Parsers
How to contribute
- submit or request the grammars/alphabet/translators you want to use!
- Get started with some issues.
- Writting adapters for other python parsing tools like pyparsing, ply or parsley
Blog posts
-
JsonSchema improvements
I have added format support to JsonSchema through pydsl. Here is a code snippet from the tests: schema = { "type" : "object", "required":["foo"], "properties" : { "foo" :...
-
random bugs in non random code
The last bug I had to fix took me 2 days to discover. I updated from python 3.2 to python 3.4. One of the tests failed randomly when it was executed with python 3.4 The error happened in the LR0 parser ====================================================================== FAIL: testArithmetic (tests.unit.test_Parser.TestLR0Parser) ---------------------------------------------------------------------- Traceback...
-
a session with pydsl 0.5.0
pydsl 0.5.0 is ready! It fixes some issues in the parser implementations and changes the way alphabet works, although the main changeset was already available in the version 0.4.1. Here is an updated session using pydsl: from pydsl.File.BNF import load_bnf_file
-
travis ci
I’ve added travisci to the project supporting python 2.7 and 3.2
-
Reviewing parsers implementations
Excluding external dependencies, PyDSL implements 4 parsers: 1.Backtracing Recusive Descent Parser It attempts every possible recursion and returns every valid parse tree. It supports the Null symbol, generates errors reports for the validator. It doesn’t support left recursion https://github.com/nesaro/pydsl/blob/master/pydsl/Parser/RecursiveDescent.py 2.Weighted Backtracing Recusive Descent Parser A small optimization...
-
Pypository project
pydsl had a repository like functionality for the main classes (Grammars, Alphabets, Functions). That code lived inside the pydsl.Memory module and provided list, search and retrieval functions. I’ve moved this code to a different project: pypository. This project can work for other projects as a repository library. The...
-
Examples from other parser libraries
I’ve added to the github repository examples from different parser libraries. Any opinions on which one is the best/most popular?
-
Pydsl Goals
1. Unifying python parsing libraries Python has a few alternatives for parsing, but some of them are unmaintained and most of them use their own format. A few examples: Ply plyplus spark Parsley Pyparsing yapps (Please visit Ned Batchelder’s Python...
-
spark library merged
I have merged a modified version of spark into. Syntax was quite old (python has evolved a lot in ten years) and I had to update a few syntax constructs, like dict.has_key, string.join or tuple expansion in function parameters. The spark integration into pydsl is still pending (see parser libraries...
-
Bounty added to pydsl
I have added a bounty for solving the task “Add parsley support” using bountysource. Hopefully this will speed things up a bit since I haven’t been able to spend time on pydsl :(.
-
Merged LR0 branch
I’ve merged LR0 branch into master. This branch includes general bugfixes in the Parser module and and LR Parser with an SLR table algorithm. This is a rewrite of the old SLR code, including support for the lexer/alphabet and symbol handling and fixing the horrible previous implementation
-
Removed function support
During the last few months I’ve been deleting old code here and there. But this time I’ve deleted one of the main components of the pydsl core: The function module. That module included: PythonFunctions Boards SyntaxDirectedTranslators ExternalProgramFunction Agents The main reason is that all those functions didn’t add...
-
Weekly update
The last two weeks I’ve been working intermittently on fixing bugs and a new release. The most important bug was related with python2/3 input handling; in python2 binaries were reading a character instead of a line. Towards the grammar aware diff, I’m refactoring old code and adding new features. The...
-
added pydsl to pypy package index
I have added pydsl to the pypy package index, so now it is possible to install pydsl with “pip install pydsl” I have also tagged commit as 0.0.1. This is not an specific release, but rather an starting point for future releases. I’ve started using github issue...
-
General diff
I’m working on a new feature to compare information according to an arbitrary grammar/alphabet. Alphabet Alphabet diff is similar to standard unix diff: using Longest Common Substring after calling the Lexer/Scanner. To some extend, this is an encoding aware diff. Grammar BNF Grammars generate a parse tree...
-
Pydsl python2 support
I’m glad to announce that all but the binaries are now python2 and python3 compatible. The only difficult bit was the metaclass support, but there are ways to make the metaclass python 2 and 3 compatible
-
pydsl_contrib repository
I’ve deleted the pydsl_contrib repository, which contained a few implementations of grammars and functions. Now it is merged into the main pydsl repository, because there was no point in having two different repositories for such a small codebase. It also added extra complexity to the installation process.
-
2 ideas and no time!
Recently I’ve had a few ideas than can improve pydsl but unfortunately I have no time to implement them… Alphabets I think that alphabets can have their own class and become part of pydsl. A lexer reads the user input (assuming and encoding) and generates a list of tokens. ...
-
Tree module refactoring
The last few days have been mildly productive in regards of coding amount and quality. The module I have been working on is pydsl.Grammar.Tree. The code is now shorter and also more concise; AST is not a class anymore. The previous class hierarchy was: Tree AST ParseTree ...
-
pydsl class diagram
From time to time I play with graphviz and python to generate class hierarchy diagrams. 2012/09/16 class diagram End of 2011 class diagram I think those diagrams illustrates very well the code cleaning and...
-
PLY support
This weekend I’ve been reading about PLY. PLY is an implementation of lex and yacc parsing tools for Python. They are usually used together in Unix environments to generate parsers. I remember using them at university, but in a very limited way (and never studying its source code)....
-
Programming languages war
Programming languages war started with the microprocessor instruction sets. Different instruction sets allows to construct the same programs, but not with the same effort. A machine contains an state(memory) and a program. The program runs and modifies the memory. The programmer decides how to modify the memory. If the memory...
-
YAGNI and pydsl
In order to improve pydsl project quality, I’ve started applying the YAGNI principle (wikipedia entry). The idea is to remove the unnecessary functionality, and in pydsl there is a lot: Exchanges and Actors: This module is not ready for use, it is not well designed and it is...
-
ANTLR grammar format support
pydsl already has a BNF-like grammar support. Standard BNF features are almost the same for all implementations Symbol1 = Symbol2 | "terminalstr" Symbol2 = "anotherterminalstr" However the advanced features are more tricky and differs heavily between implementation. pydsl supports nested grammar with terminal rules. ANTLR...
-
Python DSL
Python DSL started 4 years ago as a alternative to OO development paradigm. As a slow learner, it took me a few years to realize that what I was trying is to implement a DSL framework. One of the books I’m currently reading is Language implementation patterns by Terence...
-
Gource visualization
ColonyApps git repo visualization using gource: