Toubkal

Fully reactive programming for nodejs and the browser

Liberating your Creativity by improving your Productivity and runtime Performances

1953 Continuous Integration Tests

The current version allows rapid development of non-trivial, complex, all-reactive applications. We are using it in production applications developped for and with our clients.

Teaser

Displaying a reactive <table> which DOM container is #sales_table, ordered by date, for the years 2013 & 2014, from a source sales dataflow coming from a socket.io server, pulling the minimum amount of data from the server and updating the table as soon as some data is available from the server (complete working code including http server is available at examples/teaser):

client.js

rs.socket_io_server()
  .flow  ( 'sales' )
  .filter( [ { year: 2013 }, { year: 2014 } ] )
  .order ( [ { id: 'date' } ] )
  .table ( $( '#sales_table' ), sales_columns )
;

How does this work?

sales_table is updated reactively in realtime whenever sales are updated on the server.

rs.socket_io_server() connects the client to Toubkal socket.io server.

flow( 'sales' ) declares that the sales dataflow is needed from the server.

[ { year: 2013 }, { year: 2014 } ] is a filter query, it controls how much sales data will be pulled from the server therefore reducing both bandwidth usage and latency.

Latency is further reduced by displaying the table as soon as the first sales come from the server, improving user experience.

This query can be a dataflow, updated by DOM controls and automatically pulling the minimum amount of data from the server.

[ { id: 'date' } ] is an organizer, it can also be a dataflow dynamically updated by DOM controls, or any other application source.

The sales_columns dataflow controls table's columns. When updated, columns are reactively added or removed in realtime without any addtitional programing required. sales_columns can be defined in the client or also come from the socket.io server using the following declarative code:

var sales_columns = rs
  .socket_io_server()
  .flow( 'sales_columns' )
;

The above code automatically shares the same socket.io connection with the previous code, reducing resource usage on both clients and servers while only pulling from the server the additional sales_columns dataflow.

Table updates are optimized to add and remove the minimum set of rows and columns, improving client responsiveness, battery life and user experience.

What does this mean?

The Toubkal program above is expressed in one third the words required to express the problem in plain english replacing thousands of lines of complex and error-prone code.

Toubkal programs have no loops, and no ifs, dramatically reducing the likelyhood of bugs and hence greatly improving productivity. Under the hood, Toubkal provides all the optimized and comprehensively tested loops and ifs you need.

These same declarative techniques are applied on the server side delivering a full stack scallable and secure framework with highest performances featuring a reactive database and fine-grained authorization design patterns.

The bottom line is that Toubkal allows you to write with less code higher performance fully reactive applications, liberating your creativity.

Installation

From npm, latest release:

# npm install toubkal

Documentation

This readme provides a short introduction to Toubkal.

Full reference documentation including internals and the Toubkal protocol is available at https://toubkal.reactane.com/.

This documentation site is a Toubkal reactive application. It updates automatically after each commit is pulled on our developpement server.

Documentation is extracted from code using the following Toubkal server pipelets: - acorn(): parse javascript using the acorn library - parse_documentation(): emit documentation items from parsed comments - documentation_markdown(): format documentation items into markdown - markdown(): markdown to html converter using "markdown-it" and "highlight.js" - documentation_manuals(): Toubkal documentation manuals metadata (not content)

To work on the documentation on a local machine, run the documatation site:

  node site/server.js > site.out

Then point a web browser at localhost on port 8082.

Automated Tests, Continuous Integration

We have curently developped 1953 continuous integration tests for the Toubkal core and library pipelets that run after every commit on Travis CI under node version 8.12.0 LTS.

In the event that a test does not pass our top priority is to fix the test before anything else. We usualy fix broken tests within hours.

These tests also pass on Windows under Cygwin which although not officially supported by node, works with some caveats.

In any case, one should expect Toubkal applications to run equally well on Windows and Linux.

We also do manual testing on the following web browsers: - Chrome (latest), - Firefox (latest), - Safai (latest), - IE 11.

Running tests locally

# npm install -g coffee-script
# npm install -g mocha
# git clone https://github.com/ReactiveSets/toubkal.git
# cd toubkal
# npm install
# ./run_tests.sh
Full test results are in test.out
-> passed 1953 of 1953 tests
#
# less -R test.out # for tests detailed traces

Running core tests in a browser:

# node test/server/http.js

Then browse http://localhost:8080/test/manual/

Running browser examples:

# node examples/server.js

Then browse http://localhost:8081/

Development Stage

Toubkal is already quite reliable thanks to its comprehensive test suite and is currently used to deliver complex, all-reactive, enterprise progressive web applications for clients.

The architecture of Toubkal is now stable, with main components (pipelets, plugs, queries, transactions, and more) well defined and implemented.

Although the API may still change from time to time, we have entered a more mature phase where the high level API is now quite stable, while lower-level API changes much less often.

Our team

Toubkal is developped by a dedicated small team of experienced and passionate full-stack developers for the web and other technologies.

We are now fully funded and profitable by delivering Toubkal applications for our clients at Reactane.

We have been developping Toubkal since December 2012 and cannot imagine ever going back to programming the old way.

If you are an experienced JavaScript programmer, understand the power of all-reactive programming and would like to join our team, please contact us.

Introduction

Toubkal is a high-productivity, high-performances, scalable, all-reactive web application library aiming at improving your productivity for the development of complex applications, reducing servers' environmental footprint, and increasing mobile clients battery life by making an optimal use of server, network and client resources.

Why yet-another JavaScript application library?

The short answer is because we are not satisfied with the productivity, performances, and authorization models, of existing frameworks and libraries.

Internet servers are consuming an increasily significant share of worldwide electricity production, contributing to climate change and threatening Internet growth as the availability of cheap fosil fuels decreases due to population growth and per capita consumption growth.

The power of Internet server is now mostly increasing through the addition of CPU cores, meaning that the key to efficient usage of server resources must shift from raw-single-thread performence to high concurrency and parallelism performance. This in turn requires new programming patterns to keep, or increase programmers' productivity.

Also, one must realize that the bulk of the vast majorty of today's applications is about controling the motion of data throughout the network. Such data is no-longer limited to strictly public or strictly private informtation, requiring complex authorization schemes. This calls for new programing patterns that allows to greatly simplify the management of user authorizations well beyond all-or-nothing authentication.

What do you mean by performances?

Our first priority is high-performances, because we believe that performance is the key to better user experience, lowest operational costs, and lower environemental footprint.

We are fighting simultaneously against: - CPU cycles that consume energy to run and cool-down servers, slow-down mobile clients, and drain mobile batteries faster than anyone desires; - Latency decreasing the responsiveness of applications and user experiences; - Bandwidth usage that consume energy, and increase latency over lower-bandwidth networks.

We also want to keep good performances at scale. Most libraries either do-not-scale or scale with lower per-server performances further increasing costs while increasing environemental footprints.

Toubkal addresses all of these issues thanks to its unique Publish / Subscribe all-reactive dataflow model that works accross web browsers and nodejs servers, as well as just-in-time code generators and other unique optimizations.

Most importantly Toubkal provides a programing model that can be further optimized while maintaining user-code compatibility.

What's the big deal about authorizations?

Writing a complex application is hard-enough, add to this any significantly-complex authorization scheme and the whole thing breaks apart, slows-down to a crawl, clutters the code with plenty of unspotted security holes throughout every part of the application, and every now and then exposes end-users' data to unauthorized users.

Most companies try to get away with it by sweeping each leak under the carpet and promissing end-users that this will never happen again, or better yet, that this never happened. Internally, this usually ends-up with more meetings and paperwork, less things done for a system that although marginally improved, will at best remain unproven.

Because it is so hard, most frameworks take a this-is-not-our-problem approach to authorizations by stating that you should use third-party libraries or plugins to deal with it, all of which have shortcomming and usually will not fit the complexity of any real-world application let alone provide acceptable performances at scale.

Toubkal provides a simple yet efficient dataflow authorization model and system architecture that delivers Reactive UI updates on authorization changes at scale.

Now, you might consider that you don't need this, that end-users can refresh their page on authorization changes. But the reality is that we can do this because we provide a model that works in all cases, without requiring you to write a single additional line of code, so that you can sleep at night knowing that end-user data cannot be exposed by some piece of code that forgot to test a role in a corner-case.

How does Toubkal improve your productivity?

By allowing you to describe what you need in a declarative style, instead of how this could ever be accomplished.

Figuring-out how this should a) work securely, b) scale and c) have best performances as stated above, is hard, really hard. So hard that today the only way to achieve this is throwing millions of dollars at the problem, and/or struggling with bugs, bottlenecks and hard-to-work-around architecture limitations.

The most important concept you need to know to understand Toubkal programs is about Toubkal Pipelets and how to combine them to create programs that react to data change events:

rs.upstream_pipelet  ( parameter, ... )
  .a_pipelet         ( parameter, ... )
  .downstream_pipelet( parameter, ... )
;

A pipelet is a factory function which instances: - Maintain a dataset state, in memory, mass storage, the DOM, or virtually (stateless) - Subscribe to data change events from upstream pipelets - React to upstream events to update their dataset - Emit change events to downstream pipelets - Have no side effects on other pipelets upstream or downstream - Are piped to upstream and downstream pipelets using the '.' operator - May be connected to additional pipelets using parameters - Can be composed with other pipelets to provide new pipelets

A Toubkal program is a JavaScript program where one can mix imperative-style programming with Toubkal declarative-style programming.

Toubkal's Publish / Subscribe reactive model allows to solve the how so that you don't have to deal with it.

To make it easier, the API describes what you want in plain JavaScript without requiring a graphical UI to glue hard-coded and hard-to-comprehend xml or json "nodes" and "links" together as many other dataflow libraries require.

Toubkal reactive dataflow model provides higher level abs