Blog entries (CubicWeb's Forge) RSS Feed

CubicWeb Monthly news April 2021

François Ferry — 2021-05-12T12:09-01:00

During this period, we tried to fix issues, migrate some of our code base to latest version of CubicWeb. We also pursue some archeology tasks : merging or closing some merge requests from the old heads in CubicWeb's repository. We also release new version of logilab-database, RQL and CubicWeb!

CubicWeb 3.31, RQL 0.37 and logilab-database 1.18 are out!

We released Cubicweb 3.31, RQL 0.37 and logilab-database 1.18 on may 4th. The biggest change is the addition of two new RQL options: NULLSFIRST and NULLSLAST.

The NULLSFIRST and NULLSLAST options can be used to determine whether nulls appear before or after non-null values in the sort ordering. By default, null values sort as if larger than any non-null value; that is, NULLSFIRST is the default for DESC order, and NULLSLAST otherwise.

Furthermore, a lot of issues have been fixed, and a few new features have been implemented.

Here is an extract of the changelog:

handle same_site cookies configuration in pyramid.ini
rql: add support for order by NULLS LAST and NULLS FIRST
improve default cubicweb skeleton
dbcreate: don't ask confirmation to create schema in automatic
hooks/notification: BREAKING CHANGE correctly initialize operation with event attribute
RQLExpression: performance issue on RQLExpressions using EXISTS() BREAKING CHANGE: explicitly use EXISTS in RQLExpression for permissions
fix some security issues

One can find the full changelog on Cubicweb release page here.

Adding python types to RQL

In the past months, python types have progressively been included in RQL. This is a step forward to python types in CubicWeb itself.

A lot of work has been done, and there is still a lot of work to do. Adding types to RQL help us to simplify the refactorings. As RQL's API is not clearly defined, we will run CubicWeb's tests on each merge request of RQL, to make sure nothing breaks.

Future works

More hackathon, more developpements!

To have more time to develop CubicWeb, we will try to save a day each month to make a hackathon.

Better handling of large databases

In some CubicWeb projects, we have more and more data to manage; and databases are getting bigger and bigger. In the near future, we need to optimize the way CubicWeb deals with large databases. Table partition? More options to create indexes? We are also planning to have part of our databases checked by db experts to see what we can do.

See you next month!

CubicWeb Monthly news february/march 2021

Simon Chabot — 2021-04-09T07:11-01:00

It has been quite a time since the last public news ; we will try in the letter to summarize what we did during february and march 2021. During this period, we tried to fix issues, migrate a lot of our code base to latest version of CubicWeb. We also did some archeology ; we have been looking at all the old heads in CubicWeb's repository, and we have tried to rebase them on the latest public head. A lot of merge requests have been created and are still under review.

CubicWeb 3.30 is out!

We released Cubicweb 3.30 on march 16th. There are no “big” changes in this release. A lot of issues have been fixed, the documentation have been improved − a new tuto is coming ! − and a few new features have been implemented.

Here is an extract of the changelog:

it is now possible to use smtp authentification to send an email (#221)
required variables can be read from the environment (#85)
one can specify scripts attributes when using the add_js function (#210)
GROUP_CONCAT was not returning the expected results when NULL values were encountered (#109)
it is now possible not to drop table indexes when using the MassiveStore (#219)

One can find the full changelog on Cubicweb release page here.

Adding python types to RQL

In the past months, python types have progressively been included in RQL. This is a step forward to python types in CubicWeb itself.

Separate Back and Front

For some time now, the trend has been towards a clear separation between front and back. We are adding more features to use React in the front.

CwClientLibJS, a library allowing to use rql in the browser. CwClientLibJS is now in version 1.1.0 and is able to generate query on entity state (see !20).

react-admin-cubicweb, new tools to generate admin pages. react-admin-cubicweb is still at an early stage but can display entity attribute and relation. It also allows edition as well. The version 0.3.0 has been released !

Release-new

Release-new is one of our latest utility to release new versions of our cubes. Assuming that you are using conventional commits, you should really like this tool to release new version of your cubes.

Release-new takes care to:

update the version of the cube in __pkginfo__.py
update the debian/control if there is one
update the changelog
create a new commit with theses changes
tag the commit

By default, it will try to guess if it is a major, minor or patch release (you can specify the release type if need be). The changelog will be updated and you will be able to edit it before the commit is done. The update of the changelog will be eased if you use conventionnal commit as your commit history will be dispatched in the appropriate sections. For instance, the last changelog of CubicWeb has been produced by this tool !

Docker images

During our last hackathon, a team worked on our CubicWeb Docker images. The code to generate them has been rewritten to be simpler. During this refactoring, we took care to generate docker images for minor versions of CubicWeb as well. Therefore, on https://hub.docker.com/r/logilab/cubicweb/tags you can find docker images for major and minor versions of CubicWeb with different versions of python, to suit your needs the best we can :)

See you next month!

CubicWeb Monthly news january 2021

Fabien Amarger — 2021-02-18T09:08-01:00

New organisation

For this new year, we decided to change how we organise the CubicWeb project at Logilab. We clarified how to define priorities and how to contribute.

Rotating the coordinator of the month

First of all, we pinpointed the need to have a coordinator to organise meetings, issues and dispatch work between contributors. It was an evidence to define this role as a rotating responsability. Each month a coordinator is picked from the volontary participants. The previous coordinator and the new one have to write this monthly review together to communicate on what happened for the CubicWeb project during the month and to officially change the coordinator.

New kanban boards

To separate the on-going issues and the planned issues for the next versions, we divided the existing kanban in two.

The first one defines the ongoing work and the states of each issue. The column "IMPORTANT" is used to track the important bug which need to be fixed in the version in development.

The R&D board tracks the ideas and the future versions. This board contains an "IDEA" column, which is used as a backlog. Then we can move the "IDEA" issues into the dedicated column to plan the issue for a specific future version. The "Exploration" label is set on issues that are not asking to implement a feature, but to explore the problem and find a good implementation plan. Ideally these issues will lead to classic issues to be implemented.

The CubicWeb Forge is dead, Heptapod is our new friend

After months in transition, we decided to announce that the official site to develop CubicWeb is Logilab's Heptapod instance in replacement of the CubicWeb Forge.

Heptapod is a friendly fork of GitLab that supports Mercurial repositories, allowing to use all the features of GitLab and all the features of Mercurial (draft changesets, automatic evolution, etc).

All cubes were migrated to the new forge. CubicWeb Forge is still available, but will be made read-only and will be shutdown in the future.

Weekly meeting

Every tuesday afternoon (UTC+1), there is a weekly meeting. We discuss the issues in the kanban boards to track progress and plan the work. Feel free to join us in the CubicWeb matrix room.

CubicWeb sprint

To work on specific issues we have decided to organize regular sprints. The first one occured on January 26th and 27th 2021 with ten participants. It was a success as good work was done.

Separate Back and Front

The main subject which was studied during this sprint was the front/back separation, which is an "Exploration" issue planned for the CubicWeb v3.30.

CWClientLibJS

We discussed improving CWClientLibJS to help JavaScript developers write RQL queries into the front application. We defined a draft API for the QueryBuilder class which will be used to write RQL queries into JavaScript codebase. This API will allow to write complex RQL queries easily, such as with optionnal parameters. And the YAMS schema will be used to add helpers to get entity from eid and to organize the RQL query response depending on the entity type attributes.

The next steps will be to test this API into a project and study how to handle authentication.

CWElements

Another subject is how to generate frontend forms from the YAMS schema, such as the current CubicWeb.web implementation but all with the React framework. The result is implemented into the CWElements project: - handle relations - allow update queries.

The ongoing issues for CWElements is to test this form into a project and define how to establish a graphical identity by specifying CSS files and/or CSS framework (such as Bootstrap, Material-UI and so on)

Cleaning up the CubicWeb code repository

We corrected some bugs listed below, but more importantly we transformed all the branches from the old forge into merge requests in the new forge. Each merge request will thus be properly tracked and reviewed.

If you have a patch for CubicWeb, now is a good time to send a merge request. The merge requests will benefit from the automatimated tests. Also, all the discussion will be properly stored.

Important issues closed and merged

[v3.30] [docker] allow to read REQUIRED variables from environments #85
[v3.30] remove statsd integration? #39
[v3.30] remove web.cors in favor of wsgicors with pyramid #192
[v3.30] RQL TODAY in sqlite is not equivalent to RQL TODAY in postgresql #109
[v3.30] Include pyramid as direct deps in cubicweb by default #191
[v3.26] Box view of a CWEtype is broken #74
[v3.26 & v3.30] No translation in pviews #87

Documentation

Since we started working on Cubicweb 3.29, we have been reworking the documentation step-by-step.

A new landing page was created.

A new tutorial is being written.

It focus on:

Data import
React to display a custom page (with RQL information)
content-negociation to retrieve information in RDF format

The fictionnal usecase is a website publishing a list of museum (imported from an existing source). The final result is compiled in the tuto cube.

If you have any comment on the documentation, create an issue or come in the matrix room.

Release of CubicWeb 3.28

Simon Chabot — 2020-07-03T08:21-01:00

Hello CubicWeb community,

It is with pleasure (and some delay) that we are proud to annonce the release of CubicWeb 3.28.

The big highlights of this release are:

CubicWeb handle content negociation. You can have get entity as RDF when requested in the Accept HTTP Headers (see this commit for instance)
CubicWeb has a new dynamic database connection pooler, which replaces the old static one. (see this commit for instance).
RQL resultsets now store the variables names used in the RQL Select queries. It should ease the use of rsets and will allow to build better tools (see this commit)
CubicWeb now requires python 3.6 as a mimimum.
A big upgrade in our CI workflow has been done, both for tests and documentation.
The development of CubicWeb has moved to Logilab's heptapod forge.

To get more details about what has been added, modified or removed, you can have a look to the complete changelog published in Cubicweb's documentation.

CubicWeb 3.28 has been published :

on pipy.org as a python package.
on our debian repositories as a Debian package.
on hub.docker.com as a docker image. The tag latest now targets the 3.28 release.

CubicWeb 3.29 is now on it way. We will have tomorrow (July 3rd 2020) afternoon a v-sprint (friday-sprint) to work on the documentation of CubicWeb and its satelites. See you there !

Report of June 16th Cubicweb Meeting

Henri Cazottes — 2020-06-24T07:22-01:00

Hi everyone,

Here is the weekly report of last week meeting with some delay...

Kanban status

You can check the milestone here

Build the new cubicweb image for all the intranet apps on the public head #9
Add py{27,3}-from-forge to clients project to ensure we don't break everything when releasing #37
- MR waiting on francearchive, Laurent is going to talk to Katia about it. Also Simon says that everything is fine with 3.28rc1 on data.bnf
- all internal apps of logilab runs with 3.28rc1 and there is not bugs to signal

Todo

Setup a demo with a SPARQL API
- this work has been started few years ago but today clients are interested in this feature
- option to create an RQL-SPARQL bridge would be too long
- work in progress to compare RQL and SPARQL expressiveness, not done yet
- current lead: using rdflib-sqlalchemy by adding tables next to existing Cubicweb tables and creating a prototype of an OWL to YAMS converter
Rollback class_deprecated modifications on logilab-common
Continue typing other libs such as:
- cubicweb (complex)
- rql
- logilab-mtconverter
  - Patrick will probably start with this before continuing on RQL
- logilab-constrain
- logilab-database
Identify "good first issue" to ease contributing
Think about the documentation structure and what we want to write (for the next release)
- reduce technical debt
- spread documentation improvements among several sprints

Current work

working on fixes for YAMS tip for CW
reducing the load on the CI by removing some useless tests when triggered from other CI
adding needed commits on https://github.com/logilab/yapps to update it for python3 compatibility

See you very soon for the next report !

Report of June 10th Cubicweb Meeting

Laurent Peuch — 2020-06-10T12:46-01:00

Hi everyone,

We've just published the RC1 for CubicWeb https://pypi.org/project/cubicweb/3.28.0rc1/ and a new version 1.7.0 for logilab-common https://pypi.org/project/logilab-common/1.7.0/

Our current focus is finishing the last details for the release.

Milestone update

the changelog is now part of the documentation of logilab-common to make sure it is visible
test our clients project against our latest version on our repository to ensure we don't break everything when making a release
allow 2 randomly breaking tests to fail (those aren't part of the code we are currently working on)

Current roadmap

3.28, the version we are finishing right now, the current changelog is available here https://forge.extranet.logilab.fr/cubicweb/cubicweb/blob/branch/default/doc/changes/3.28.rst
3.29 on which we'll focus on writing documentation
4.0 in which we'll introduce the usage of semver for CubicWeb regarding its dependencies and release YAMS refactored version

Semver

One of our focus right now is to make stable releases of our core projects that won't break all the things ™ and we've made a lot of improvement in our testing suit to ensure that we test everything against our latest modifications before a release is made. Another problem we have right now is that CW only depends on a minimum version number for its dependencies, this mean that if we want to make a new release for one of the dependencies that will have some breaking code this introduce the risk of breaking all new CW installations.

To solve this situation we have decided to implement semantic versioning and only introduction breaking changes in major releases and in addition to only depends on one specific major release at the time in CW dependencies. This way, when we need to make a new release with breaking changes, this will be a major release and we won't break all new CW installations.

We have planned to start implementing this strategy starting CW version 4.0

Various updates

a lot of fixes have been pushed on YAMS and CubicWeb to make CubicWeb compatible with the latest modifications in YAMS

See you next week!

Report of June 3rd Cubicweb Meeting

Henri Cazottes — 2020-06-03T13:11-01:00

Hi everyrone,

Version 3.28-rc1 is on its way! First, let's have a look to the issue board state.

Milestone update

Introduced types #10
- logilab.common.deprecation has been typed (see hackathon report below): done
Add tests for the content negociation !20: MR about to be accepted
Update logilab-common changelogs #43 : done
Add automatic doc re-build to the CubicWeb CI #8 : done

Todo

Review and accept MR !20
Release logilab-common and cubicweb 3.28-rc1

Semver discussions

Right now, dependencies are only specifying a minimal version. So if we introduce a breaking change in a new version, apps might break too. We plan to follow semver convention to prevent this from happening.

We also discussed the idea of aligning version between compatible tools, so every major version would work with the same major version of other tools/dependencies.

This idea will be introduced in 3.29 documentation, but will probably start with the release of Cubicweb version 4.

Hackathon

Last Friday we did an internal hackathon at Logilab and Laurent, Noé and I spent time working on Cubicweb. We mainly:

wrote changelogs for:
- logilab-common
- cubicweb
tried to add a Merge Request template on Cubicweb
- doesn't work on Heptapod actually, we will ask Octobus to have a look (see #46)
added annotation types on logilab.common.deprecated
improved tox.ini and added a gitlag-ci.yaml file in cube skeleton

That's all! You should receive and email soon about the rc1 release.

Thanks for reading,

Henri

Report of May 26th Cubicweb Meeting

Henri Cazottes — 2020-06-02T07:23-01:00

Hi,

Welcome back to another weekly report! Today, the following topics have been discussed.

Broken tests situation, follow up

Migrating CubicWeb to Heptapod and modifications in dependencies resulted in broken tests as it was presented last week. Work has been done on Friday afternoon thanks to Simon and Laurent, but it's not fixed yet. Tox is now happy but we still have a bug on a test that succeeds locally but not when run by the CI job. We do have a lead which may concern firefox usage in headless mode. Jobs logs are available here.

Milestone update

Introduced types
- Types have been added in Yams, merge request about to be reviewed
- We choose to dissociate this issue from the actual release as it's more related to Yams and not mandatory to release a 3.28 version.
- Re-build ReadTheDoc for every release
- Done for most of the dependencies but not CubicWeb yet
- Two dependencies, mtconvert and constraint don't have doc nor tests, we think about removing them instead of creating and maintaining this code which is pretty old
Move to semantic versionning
- we talked about improving dependencies requirements to ease the release process (not giving only one version)
- we think we should stick to semver, but we need to discuss it more (should we bump all major version to the same number for interoperable dependencies, etc...)
- As those questions need to be discussed, we choose to move this issue to the 3.29 milestone
Add tests to content negociation
- Need to be done
Check if ?vid=rdf is still working
- Done
- We did spot that requesting vid=rdf or using content negotiation would not return the same RDF. We should fix this to have a more consistent behavior. Will be added for the next release.

Todo before releasing version 3.28

To sum up, before releasing the next CubicWeb version, we need to:

Fix CI tests
Add automatic doc re-build to the CubicWeb CI

This should be done and released within the next two weeks.

Side notes

For the next release we should align CubicWeb with changes made in Yams
Release early, release often

See you next week!

Report of May 19th Cubicweb Meeting

Simon Chabot — 2020-05-20T15:41-01:00

Hi everyone,

Yesterday we held a new CubicWeb meeting, and we would like to share with you what was said during that meeting :

some issues have been migrated from cubicweb.org forge to the heptapod forge. You can find them here. Only the issues which were in the cubicweb-dev-board have been migrated, the others have been considered too old to make their migration worthwhile ;
new milestones have been created, and some issues have been affected to them ;
if you want to help out with Cubicweb's development, you can have a look to the Issues Board, and pick one task in the “To Do” column (this task should be related to a milestone) ;
CW tests are still failing on the forge, and it's also related to other packages that have been released. Fixing those tests is quite urgent now, therefore we suggest to fix them in a sprint this Friday afternoon. Feel
free to join !
On main Cubicweb's dependencies (RQL, YAMS, logilab-common, etc), a heptapod job has been added to trigger CW's tests with the last version of the dependency in the forge and not only the last released version on pypi. This should help to release new versions of CW's dependencies with
more confidence. (for now, it only triggers the job, a future version of heptapod should provide a better integration of multi-project pipelines) ;
The documentations of CubicWeb's dependencies are now automatically published on readthedocs. The work is in progress for CubicWeb itself ;

Next week, if the tests are successful, we will talk about a release candidate for 3.28.

Stay tuned :)

A roadmap to Cubicweb 3.28 (and beyond)

Simon Chabot — 2020-05-13T08:03-01:00

Yesterday at Logilab we had a small meeting to discuss about a roadmap to Cubicweb 3.28 (and beyond), and we would like to report back to you from this meeting.

Cubicweb 3.28 will mainly bring the implementation of content negotiation. It means that Cubicweb will handle content negotiation and will be able to return RDF using Cubicweb's ontology when requested by a client.

The 3.28 will have other features as well (like a new variables attributes to ResultSet that contains the name of the projected variables, etc). Those features will be detailed the release changelog.

Before releasing this version, we would like to finish the migration to heptapod, to make sure that everything is ok. The remaining tasks are:

fixing the CI (there is still some random failings, that need further investigation)
migrate the jenkins job that pushes images to hub.docker.com on heptapod, to make everthing available from the forge. It will be explicit for everyone when a job is done, and what is its status.

Beside of releasing Cubicweb 3.28, its ecosystem will also be updated:

logilab-common, a new version will be released very soon, which brings a refactoring of the deprecation system, and annotations (coming from pyannotate)
yams, a new version is coming. This version:
brings type annotation (manually done, a carefully checked);
removes a lot of abbreviation to make the code clearer;
removes some magic related to a object which used to behave like a string;

The goal of these two releases, is to have type annotations in the core libraries used by CubicWeb, and then to be able to bring type annotation into CubicWeb itself, in a future version.

On those projects, some “modernisation” has been started too ; (fixing flake8 when needed, repaint the code black). This “modernisation” step is still on going on the different projects related to CubicWeb (and achieved for yams, and logilab-common).

In the medium term, we would like to focus on the documentation of CubicWeb and its ecosystem. We do know that it's really hard for newcomers (and even ourself sometime) to understand how to start, what each module is doing etc. An automatic documentation has been released for some modules (see 1, 2 or 3 for instance). It would be nice to automatize the update of the documentation on readthedocs, update the old examples, and add new ones about the new feature we are adding (like content negotiation, pyramid predicates, etc). This could be done in team Friday's sprint or hackathon for instance. CubicWeb would also need some modernisation (running black ? and above all, make all files flake8 compilant…).

Regarding CubicWeb development, all (or, at least a lot of) cubes and Cubicweb related projects moved from cubicweb.org's forge to our instance of heptapod (4 and 5). Some issues have been imported from cubicweb.org to heptapod. New issues should be opened on heptapod, and the review should also be done there. We hope that will ease the reappropriation of the code basis and stimulates new merge-requests :)

To end this report, we like to emphasis that we will try to make a « remote Cubicweb meeting » each Tuesday at 2 pm. If you would like to participate to this meeting, it's with great pleasure (if you need the webconference URL, contact one of us, we will provide it to you). We also created a #Cubicweb channel on matrix.logilab.org ; feel free to ask for an invitation if you'd like to discuss Cubicweb related things with us.

All the best, and… see you next Tuesday :)

What is new in CubicWeb 3.27 ?

Nicolas Chauvat — 2020-02-03T08:55-01:00

Hello CubicWeb community,

We are pleased to announce the release of CubicWeb 3.27. Many thanks to all the contributors of this release!

Main changes in this release are listed below. Please note this release drops python2 support.

Enjoy this new version!

New features

Tests can now be run concurrently across multiple processes. You can use pytest-xdist for that. For tests using PostgresApptestConfiguration you should be aware that startpgcluster() can't run concurrently. Workaround is to call pytest with --dist=loadfile to use a single test process per test module or use an existing database cluster and set db-host and db-port of devtools.DEFAULT_PSQL_SOURCES['system'] accordingly.

on cubicweb-ctl create and cubicweb-ctl pyramid, if it doesn't already exist in the instance directory, the pyramid.ini file will be generated with the needed secrets.
add a --pdb flag to all cubicweb-ctl command to launch (i)pdb if an exception occurs during a command execution.
the --loglevel and --dbglevel flags are available for all cubicweb-ctl instance commands (and not only the pyramid one)
following "only in foreground" behavior all commands logs to stdout by default from now on. To still log to a file pass log_to_file=True to CubicWebConfiguration.config_for
add a new migration function update_bfss_path(old_path, new_path) to update the path in Bytes File-System Storage (bfss).
on every request display request path and selected controller in CLI
migration interactive mode improvements:
- when an exception occurs, display the full traceback instead of only the exception
- on migration p(db) choice, launch ipdb if it's installed
- on migration p(db) choice, give the traceback to pdb if it's available, this mean that the (i)pdb interactive session will be on the stack of the exception instead of being on the stack where pdb is launched which will allow the user to access all the relevant context of the exception which otherwise is lost
on DBG_SQL and/or DBG_RQL, if pygments is installed, syntax highlight sql/rql debug output
allow to specify the instance id for any instance command using the CW_INSTANCE global variable instead of or giving it as a cli argument
when debugmode is activated ('-D/--debug' on the pyramid command for example), the HTML generated by CW will contains new tags that will indicate by which object in the code it has been generated and in which line of which source code. For example:

<div
  cubicweb-generated-by="cubicweb.web.views.basetemplates.TheMainTemplate"
  cubicweb-from-source="/home/user/code/logilab/cubicweb/cubicweb/web/views/basetemplates.py:161"
  id="contentmain">
    <h1
      cubicweb-generated-by="cubicweb.web.views.basetemplates.TheMainTemplate"
      cubicweb-from-source="/home/user/code/logilab/cubicweb/cubicweb/view.py:136">
        unset title
    </h1>
    [...]
</div>

While this hasn't been done yet, this feature is an open path for building dynamic tools that can help inspect the page.

a new debug channels mechanism has been added, you can subscribe to one of those channels in your python code to build debug tools for example (the pyramid custom panels are built using that) and you will receive a datastructure (a dict) containing related information. The available channels are: controller, rql, sql, vreg, registry_decisions
add a new '-t/--toolbar' option the pyramid command to activate the pyramid debugtoolbar
a series of pyramid debugtoolbar panels specifically made for CW, see bellow

Pyramid debugtoolbar and custom panel

The pyramid debugtoolbar is now integrated into CubicWeb during the development phase when you use the 'pyramid' command. To activate it you need to pass the '-t/--toolbar' argument to the 'pyramid' command.

In addition, a series of custom panels specifically done for CW are now available, they display useful information for the development and the debugging of each page. The available panels are:

a general panel which contains the selected controller, the current settings and useful links screenshot1

a panel listing all decisions taken in registry for building this page screenshot2

a panel listing the content of the vreg registries screenshot3

a panel listing all the RQL queries made during a request screenshot4

a panel listing all the SQL queries made during a request screenshot5

Furthermore, in all those panels, next to each object/class/function/method a link to display its source code is available (shown as '[source]' screenshot6) and also every file path shown is a traceback is also a link to display the corresponding file (screenshot7). For example: screenshot8.

Backwards incompatible changes

Standardization on the way to launch a cubicweb instance, from now on the only way to do that will be the used the pyramid command. Therefore:
- cubicweb-ctl commands "start", "stop", "restart", "reload" and "status" have been removed because they relied on the Twisted web server backend that is no longer maintained nor working with Python 3.
- Twisted web server support has been removed.
- cubicweb-ctl wsgi has also been removed.
Support for legacy cubes (in the 'cubes' python namespace) has been dropped. Use of environment variables CW_CUBES_PATH and CUBES_DIR is removed.
Python 2 support has been dropped.
Exceptions in notification hooks aren't catched-all anymore during tests so one can expect tests that seem to pass (but were actually silently failing) to fail now.
All "cubicweb-ctl" command only accept one instance argument from now one (instead of 0 to n)
'pyramid' command will always run in the foreground now, by consequence the option --no-daemon has been removed.
DBG_MS flag has been removed since it is not used anymore
transactions db logs where displayed using the logging (debug/info/warning...) mechanism, now it is only displayed if the corresponding DBG_OPS flag is used

Deprecated code drops

Most code deprecated by version 3.25 or older versions has been dropped.

Implementing the langserver protocol for RQL

Laurent Peuch — 2019-10-31T14:51-01:00

One of our next project for cubicweb and its ecosystem is to implement the langserver protocol for the RQL language that we are using to query the data stored in CubicWeb. The langserver protocol is an idea to solve one problem: to integrate operation for various languages, most IDE/tools needs to reimplement the wheel all the time, doing custom plugin etc... To solve this issue, this protocol has been invented with one idea: make one server for a language, then all IDE/tools that talks this protocol will be able to integrate it easily.

So the idea is simple: let's build our own server for RQL so we'll be able to integrate it everywhere and build tools for it.

Since RQL has similarities with GraphQL, one of the goals is to have something similar to Graphiql which is for example used by GitHub to expose their API at https://developer.github.com/v4/explorer/

So this post has several objectives:

gather people that would be motivate to work on that subject, for now there is Laurent Wouters and me :)
explain to you in more details (not all) how the language server protocol works
show what is already existing for both langserver in python and rql
show the first roadmap we've discussed with Laurent Wouters on how we think we can do that :)
be a place to discuss this project, things aren't fixed yet :)

So, what is the language server protocol (LSP)?

It's a JSON-RPC based protocol where the IDE/tool talks to the server. JSON-RPC, said simply, is a bi-directional protocol in json.

In this procotol you have 2 kind of exchanges:

requests: where the client (or server) ask the server (or the server ask the client) something and a reply is expected. For example: where is the definition of this function?
notifications: the same but without an expected reply. For example: linting information or error detection

The LSP specifications has 3 bigs categories:

everything about initialization/shutdown the server etc...
everything regarding text and workspace synchronization between the server and the client
the actual things that interest us: a list of languages features that the server supports (you aren't in the obligation to implement everything)

Here is the simplified list of possible languages features that the website present:

Code completion
Hover
Jump to def
Workspace symbols
Find references
Diagnostics

The specification is much more detailed but way less comprehensive (look at the "language features" on the right menu for more details):

completion/completion resolve
hover (when you put your cursor on something)
signatureHelp
declaration (go to...)
definition (go to...)
typeDefinition (go to...)
implementation (go to...)
references
documentHighlight (highlight all references to a symbol)
documentSymbol ("symbol" is a generic term for variable, definitions etc...)
codeAction (this one is interesting)
codeLens/codeLens resolve
documentLink/documentLink resolve
documentColor/colorPresentation (stuff about picking colors)
formatting/rangeFormatting/onTypeFormatting (set tab vs space)
rename/prepareRename
foldingRange

(Comments are from my current understanding of the spec, it might not be perfect)

The one that is really interesting here (but not our priority right now) is "codeAction", it's basically a generic entry point for every refactoring kind of operations as some examples from the spec shows:

Example extract actions:

Extract method

Extract function

Extract variable

Extract interface from class

Example inline actions:

Inline function

Inline variable

Inline constant

Example rewrite actions:

Convert JavaScript function to class

Add or remove parameter

Encapsulate field

Make method static

Move method to base class

But I'm not expecting us to have direct need for it but that really seems one to keep in mind.

One question that I frequently got was: is syntax highlight included in the langserver protocol? Having double checked with Laurent Wouters, it's actually not the case (I thought documentSymbol could be used for that but actually no).

But we already have an implementation for that in pygments: https://hg.logilab.org/master/rql/file/d30c34a04ebf/rql/pygments_ext.py

What is currently existing for LSP in python and rql

The state is not great in the python ecosystem but not a disaster. Right now I haven't been able to find any generic python implementation of LSP that we could really reuse and integrate.

There is, right now and to my knowledge, only 2 maintained implementation of LSP in python. One for python and one for ... Fortran x)

Palantir's one makes extensive use of advanced magic code doesn't seems really necessary but it is probably of higher quality code since the Fortran one doesn't seems very idiomatic but looks much simpler.

So we'll ever need to extract the needed code from one of those of implement our own, not so great.

On the RQL side, everything that seems to be useful for our current situation is located in the RQL package that we maintain: https://hg.logilab.org/master/rql

Roadmap

After a discussion with Laurent Wouters, a first roadmap looks like this:

extract the code from either palantir or fortran LSP implementation and come with a generic implementation (I'm probably going to do it but Laurent told me he his going to take a look too) When I'm talking about a generic implementation I'm talking about everything listed in the big category of the protocol that isn't related to language features which we don't really want to rewrite again.

Once that's done, start implementing the language features for RQL:

the easiest is the syntax errors detection code, we just need to launch to parser on the code and handle the potential errors
do that with pretty specific red underline
play with RQL AST to extract the symbols and start doing things like codeLens and hover
much more complex (and for later): autocompletion (we'll either need a demi compiler or to modify the current one for that)

Side note

To better understand the motivation behind this move, it is part of the more global move of drop the "Web" from CubicWeb and replace all the front end current implementation by reactjs+typescript views. In this context CubicWeb (or Cubic?) will only serves as a backend provide with which we will talk in... RQL! Therefor writing and using RQL will be much more important than right now.

Hypermedia API with cubicweb-jsonschema

Denis Laxalde — 2017-04-04T12:20-01:00

This is the second post of a series about cubicweb-jsonschema. The first post mainly dealt with JSON Schema representations of CubicWeb entities along with a brief description of the JSON API. In this second post, I'll describe another aspect of the project that aims at building an hypermedia API by leveraging the JSON Hyper Schema specification.

Hypermedia APIs and JSON Hyper Schema

Hypermedia API is somehow a synonymous of RESTful API but it makes it clearer that the API serves hypermedia responses, i.e. content that helps discoverability of other resources.

At the heart of an hypermedia API is the concept of link relation which both aims at describing relationships between resources as well as provinding ways to manipulate them.

In JSON Hyper Schema terminology, link relations take the form of a collection of Link Description Objects gathered into a links property of a JSON Schema document. These Link Description Objects thus describes relationships between the instance described by the JSON Schema document at stake and other resources; they hold a number of properties that makes relationships manipulation possible:

rel is the name of the relation, it is usually one of relation names registered at IANA;
href indicates the URI of the target of the relation, it may be templated by a JSON Schema;
targetSchema is a JSON Schema document (or reference) describing the target of the link relation;
schema (recently renamed as submissionSchema) is a JSON Schema document (or reference) describing what the target of the link expects when submitting data.

Hypermedia walkthrough

In the remaining of the article, I'll walk through a navigation path that is made possible by hypermedia controls provided by cubicweb-jsonschema. I'll continue on the example application described in the first post of the series which schema consists of Book, Author and Topic entity types. In essence, this walkthrough is typical of what an intelligent client could do when exposed to the API, i.e. from any resource, discover other resources and navigate or manipulate them.

This walkthrough assumes that, given any resource (i.e. something that has a URL like /book/1), the server would expose data at the main URL when the client asks for JSON through the Accept header and it would expose the JSON Schema of the resource at a schema view of the same URL (i.e. /book/1/schema). This assumption can be regarded as a kind of client/server coupling, which might go away in later implementation.

Site root

While client navigation could start from any resource, we start from the root resource and retrieve its schema:

GET /schema
Accept: application/schema+json

HTTP/1.1 200 OK
Content-Type: application/json

{
    "links": [
        {
            "href": "/author/",
            "rel": "collection",
            "schema": {
                "$ref": "/author/schema?role=creation"
            },
            "targetSchema": {
                "$ref": "/author/schema"
            },
            "title": "Authors"
        },
        {
            "href": "/book/",
            "rel": "collection",
            "schema": {
                "$ref": "/book/schema?role=creation"
            },
            "targetSchema": {
                "$ref": "/book/schema"
            },
            "title": "Books"
        },
        {
            "href": "/topic/",
            "rel": "collection",
            "schema": {
                "$ref": "/topic/schema?role=creation"
            },
            "targetSchema": {
                "$ref": "/topic/schema"
            },
            "title": "Topics"
        }
    ]
}

So at root URL, our application serves a JSON Hyper Schema that only consists of links. It has no JSON Schema document, which is natural since there's usually no data bound to the root resource (think of it as empty rset in CubicWeb terminology).

These links correspond to top-level entity types, i.e. those that would appear in the default startup page of a CubicWeb application. They all have "rel": "collection" relation name (this comes from RFC6573) as their target is a collection of entities. We also have schema and targetSchema properties.

Links as actions

So let's pick one of these links, the Books link. We can either just follow the link URI href="/book/" using a GET request, but what if we wanted to create a new book? We'd need to perform a POST request at href URI with a body that conforms to the JSON Schema of schema property. So let's first retrieve this schema:

GET /book/schema?role=creation
Accept: application/schema+json

HTTP/1.1 200 OK
Content-Length: 836
Content-Type: application/json
Date: Tue, 04 Apr 2017 09:46:53 GMT
Server: waitress

{
    "$ref": "#/definitions/Book",
    "definitions": {
        "Book": {
            "additionalProperties": false,
            "properties": {
                "author": {
                    "items": {
                        "oneOf": [
                            {
                                "enum": [
                                    "856"
                                ],
                                "title": "Ernest Hemingway"
                            },
                            {
                                "enum": [
                                    "855"
                                ],
                                "title": "Victor Hugo"
                            }
                        ],
                        "type": "string"
                    },
                    "maxItems": 1,
                    "minItems": 1,
                    "title": "author",
                    "type": "array"
                },
                "publication_date": {
                    "format": "date-time",
                    "title": "publication date",
                    "type": "string"
                },
                "title": {
                    "title": "title",
                    "type": "string"
                }
            },
            "required": [
                "title",
                "publication_date"
            ],
            "title": "Book",
            "type": "object"
        }
    }
}

(This is essentially the same schema that we had in the last example of the first post.)

With this we may form a POST request with a JSON document as body that conforms to this schema, but how do we know that we can actually send the request? Hypermedia link relations do not provide any information for this. Instead, we rely on the underlying protocol (HTTP) and on the server to advertize its capabilities. This means we need to fetch the target ressource and inspect the Allow header in the response. Most of the times, we would already have fetched the resource (verb GET) so these headers would be readily available. But in this case, we are on the root resource and we have not fetched targets of the links, so we use a HEAD request:

HEAD /book/

HTTP/1.1 200 OK
Allow: GET, POST

Verbs listed in this Allow header just describe what actions we are allowed to perform on the resource. Under the hood, this is determined by permissions lookup (i.e. "read" permission matches with GET verb, "add" permission matches with POST verb and so on).

So we can create a Book!

POST /book/
Accept: application/json

{"title": "L'homme qui rit", "author": ["855"], "publication_date": "1869-04-01"}

HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8
Location: http://localhost:6543/Book/859

{
    "author": [
        "Victor Hugo"
    ],
    "publication_date": "1869-04-01",
    "title": "L'homme qui rit"
}

What's important in this response is the Location header which indicates where our new resource lives. Our intelligent client can then proceed with its navigation using this information.

From collection to items

Now that we have added a new book, let's step back and use our books link to retrieve data (verb GET):

GET /book/
Accept: application/json

HTTP/1.1 200 OK
Allow: GET, POST
Content-Type: application/json

[
    {
        "id": "859",
        "title": "L'homme qui rit"
    },
    {
        "id": "858",
        "title": "The Old Man and the Sea"
    },
]

which, as always, needs to be completed by a JSON Schema:

GET /book/schema
Accept: application/schema+json


HTTP/1.1 200 OK
Content-Type: application/json

{
    "$ref": "#/definitions/Book_plural",
    "definitions": {
        "Book_plural": {
            "items": {
                "properties": {
                    "id": {
                        "type": "string"
                    },
                    "title": {
                        "type": "string"
                    }
                },
                "type": "object"
            },
            "title": "Books",
            "type": "array"
        }
    },
    "links": [
        {
            "href": "/book/",
            "rel": "collection",
            "schema": {
                "$ref": "/book/schema?role=creation"
            },
            "targetSchema": {
                "$ref": "/book/schema"
            },
            "title": "Books"
        },
        {
            "href": "/book/{id}",
            "rel": "item",
            "targetSchema": {
                "$ref": "/book/schema?role=view"
            },
            "title": "Book"
        }
    ]
}

Consider the last item of links in the above schema. It has a "rel": "item" property which indicates how to access items of the collection; its href property is a templated URI which can be expanded using instance data and schema (here we only have a single id template variable).

So our client may navigate to the first item of the collection (id="859") at /book/859 URI, and retrieve resource data:

GET /book/859
Accept: application/json

HTTP/1.1 200 OK
Allow: GET, PUT, DELETE
Content-Type: application/json

{
    "author": [
        "Victor Hugo"
    ],
    "publication_date": "1869-04-01T00:00:00",
    "title": "L'homme qui rit"
}

and schema:

GET /book/859/schema
Accept: application/schema+json

HTTP/1.1 200 OK
Content-Type: application/json

{
    "$ref": "#/definitions/Book",
    "definitions": {
        "Book": {
            "additionalProperties": false,
            "properties": {
                "author": {
                    "items": {
                        "type": "string"
                    },
                    "title": "author",
                    "type": "array"
                },
                "publication_date": {
                    "format": "date-time",
                    "title": "publication date",
                    "type": "string"
                },
                "title": {
                    "title": "title",
                    "type": "string"
                },
                "topics": {
                    "items": {
                        "type": "string"
                    },
                    "title": "topics",
                    "type": "array"
                }
            },
            "title": "Book",
            "type": "object"
        }
    },
    "links": [
        {
            "href": "/book/",
            "rel": "up",
            "targetSchema": {
                "$ref": "/book/schema"
            },
            "title": "Book_plural"
        },
        {
            "href": "/book/859/",
            "rel": "self",
            "schema": {
                "$ref": "/book/859/schema?role=edition"
            },
            "targetSchema": {
                "$ref": "/book/859/schema?role=view"
            },
            "title": "Book #859"
        }
    ]
}

Entity resource

The resource obtained above as an item of a collection is actually an entity. Notice the rel="self" link. It indicates how to manipulate the current resource (i.e. at which URI, using a given schema depending on what actions we want to perform). Still this link does not indicate what actions may be performed. This indication is found in the Allow header of the data response above:

Allow: GET, PUT, DELETE

With these information bits, our intelligent client is able to, for instance, form a request to delete the resource. On the other hand, the action to update the resource (which is allowed because of the presence of PUT in Allow header, per HTTP semantics) would take the form of a request which body conforms to the JSON Schema pointed at by the schema property of the link.

Also note the rel="up" link which makes it possible to navigate to the collection of books.

Conclusions

This post introduced the main hypermedia capabilities of cubicweb-jsonschema, built on top of the JSON Hyper Schema specification. The resulting Hypermedia API makes it possible for an intelligent client to navigate through hypermedia resources and manipulate them by using both link relation semantics and HTTP verbs.

In the next post, I'll deal with relationships description and manipulation both in terms of API (endpoints) and hypermedia representation.

Introducing cubicweb-jsonschema

Denis Laxalde — 2017-03-23T08:57-01:00

This is the first post of a series introducing the cubicweb-jsonschema project that is currently under development at Logilab. In this post, I'll first introduce the general goals of the project and then present in more details two aspects about data models (the connection between Yams and JSON schema in particular) and the basic features of the API. This post does not always present how things work in the current implementation but rather how they should.

Goals of cubicweb-jsonschema

From a high level point of view, cubicweb-jsonschema addresses mainly two interconnected aspects. One related to modelling for client-side development of user interfaces to CubicWeb applications while the other one concerns the HTTP API.

As far as modelling is concerned, cubicweb-jsonschema essentially aims at providing a transformation mechanism between a Yams schema and JSON Schema that is both automatic and extensible. This means that we can ultimately expect that Yams definitions alone would sufficient to have generated JSON schema definitions that would consistent enough to build an UI, pretty much as it is currently with the automatic web UI in CubicWeb. A corollary of this goal is that we want JSON schema definitions to match their context of usage, meaning that a JSON schema definition would not be the same in the context of viewing, editing or relationships manipulations.

In terms of API, cubicweb-jsonschema essentially aims at providing an HTTP API to manipulate entities based on their JSON Schema definitions.

Finally, the ultimate goal is to expose an hypermedia API for a CubicWeb application in order to be able to ultimately build an intelligent client. For this we'll build upon the JSON Hyper-Schema specification. This aspect will be discussed in a later post.

Basic usage as an HTTP API library

Consider a simple case where one wants to manipulate entities of type Author described by the following Yams schema definition:

class Author(EntityType):
    name = String(required=True)

With cubicweb-jsonschema one can get JSON Schema for this entity type in at different contexts such: view, creation or edition. For instance:

in a view context, the JSON Schema will be:

{
    "$ref": "#/definitions/Author",
    "definitions": {
        "Author": {
            "additionalProperties": false,
            "properties": {
                "name": {
                    "title": "name",
                    "type": "string"
                }
            },
            "title": "Author",
            "type": "object"
        }
    }
}

whereas in creation context, it'll be:

{
    "$ref": "#/definitions/Author",
    "definitions": {
        "Author": {
            "additionalProperties": false,
            "properties": {
                "name": {
                    "title": "name",
                    "type": "string"
                }
            },
            "required": [
                "name"
            ],
            "title": "Author",
            "type": "object"
        }
    }
}

(notice, the required keyword listing name property).

Such JSON Schema definitions are automatically generated from Yams definitions. In addition, cubicweb-jsonschema exposes some endpoints for basic CRUD operations on resources through an HTTP (JSON) API. From the client point of view, requests on these endpoints are of course expected to match JSON Schema definitions. Some examples:

Get an author resource:

GET /author/855
Accept:application/json

HTTP/1.1 200 OK
Content-Type: application/json
{"name": "Ernest Hemingway"}

Update an author:

PATCH /author/855
Accept:application/json
Content-Type: application/json
{"name": "Ernest Miller Hemingway"}

HTTP/1.1 200 OK
Location: /author/855/
Content-Type: application/json
{"name": "Ernest Miller Hemingway"}

Create an author:

POST /author
Accept:application/json
Content-Type: application/json
{"name": "Victor Hugo"}

HTTP/1.1 201 Created
Content-Type: application/json
Location: /Author/858
{"name": "Victor Hugo"}

Delete an author:

DELETE /author/858

HTTP/1.1 204 No Content

Now if the client sends invalid input with respect to the schema, they'll get an error:

(We provide a wrong born property in request body.)

PATCH /author/855
Accept:application/json
Content-Type: application/json
{"born": "1899-07-21"}

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
    "errors": [
        {
            "details": "Additional properties are not allowed ('born' was unexpected)",
            "status": 422
        }
    ]
}

From Yams model to JSON Schema definitions

The example above illustrates automatic generation of JSON Schema documents based on Yams schema definitions. These documents are expected to help developping views and forms for a web client. Clearly, we expect that cubicweb-jsonschema serves JSON Schema documents for viewing and editing entities as cubicweb.web serves HTML documents for the same purposes. The underlying logic for JSON Schema generation is currently heavily inspired by the logic of primary view and automatic entity form as they exists in cubicweb.web.views. That is: the Yams schema is introspected to determine how properties should be generated and any additionnal control over this can be performed through uicfg declarations [1].

To illustrate let's consider the following schema definitions which:

class Book(EntityType):
    title = String(required=True)
    publication_date = Datetime(required=True)

class Illustration(EntityType):
    data = Bytes(required=True)

class illustrates(RelationDefinition):
    subject = 'Illustration'
    object = 'Book'
    cardinality = '1*'
    composite = 'object'
    inlined = True

class Author(EntityType):
    name = String(required=True)

class author(RelationDefinition):
    subject = 'Book'
    object = 'Author'
    cardinality = '1*'

class Topic(EntityType):
    name = String(required=True)

class topics(RelationDefinition):
    subject = 'Book'
    object = 'Topic'
    cardinality = '**'

and consider, as before, JSON Schema documents in different contexts for the the Book entity type:

in view context:

{
    "$ref": "#/definitions/Book",
    "definitions": {
        "Book": {
            "additionalProperties": false,
            "properties": {
                "author": {
                    "items": {
                        "type": "string"
                    },
                    "title": "author",
                    "type": "array"
                },
                "publication_date": {
                    "format": "date-time",
                    "title": "publication_date",
                    "type": "string"
                },
                "title": {
                    "title": "title",
                    "type": "string"
                },
                "topics": {
                    "items": {
                        "type": "string"
                    },
                    "title": "topics",
                    "type": "array"
                }
            },
            "title": "Book",
            "type": "object"
        }
    }
}

We have a single Book definition in this document, in which we find attributes defined in the Yams schema (title and publication_date). We also find the two relations where Book is involved: topics and author, both appearing as a single array of "string" items. The author relationship appears like that because it is mandatory but not composite. On the other hand, the topics relationship has the following uicfg rule:

uicfg.primaryview_section.tag_subject_of(('Book', 'topics', '*'), 'attributes')

so that it's definition appears embedded in the document of Book definition.

A typical JSON representation of a Book entity would be:

{
    "author": [
        "Ernest Miller Hemingway"
    ],
    "title": "The Old Man and the Sea",
    "topics": [
        "sword fish",
        "cuba"
    ]
}

in creation context:

{
    "$ref": "#/definitions/Book",
    "definitions": {
        "Book": {
            "additionalProperties": false,
            "properties": {
                "author": {
                    "items": {
                        "oneOf": [
                            {
                                "enum": [
                                    "855"
                                ],
                                "title": "Ernest Miller Hemingway"
                            },
                            {
                                "enum": [
                                    "857"
                                ],
                                "title": "Victor Hugo"
                            }
                        ],
                        "type": "string"
                    },
                    "maxItems": 1,
                    "minItems": 1,
                    "title": "author",
                    "type": "array"
                },
                "publication_date": {
                    "format": "date-time",
                    "title": "publication_date",
                    "type": "string"
                },
                "title": {
                    "title": "title",
                    "type": "string"
                }
            },
            "required": [
                "title",
                "publication_date"
            ],
            "title": "Book",
            "type": "object"
        }
    }
}

notice the differences, we now only have attributes and required relationships (author) in this schema and we have the required listing mandatory attributes; the author property is represented as an array which items consist of pre-existing objects of the author relationship (namely Author entities).

Now assume we add the following uicfg declaration:

uicfg.autoform_section.tag_object_of(('*', 'illustrates', 'Book'), 'main', 'inlined')

the JSON Schema for creation context will be:

{
    "$ref": "#/definitions/Book",
    "definitions": {
        "Book": {
            "additionalProperties": false,
            "properties": {
                "author": {
                    "items": {
                        "oneOf": [
                            {
                                "enum": [
                                    "855"
                                ],
                                "title": "Ernest Miller Hemingway"
                            },
                            {
                                "enum": [
                                    "857"
                                ],
                                "title": "Victor Hugo"
                            }
                        ],
                        "type": "string"
                    },
                    "maxItems": 1,
                    "minItems": 1,
                    "title": "author",
                    "type": "array"
                },
                "illustrates": {
                    "items": {
                        "$ref": "#/definitions/Illustration"
                    },
                    "title": "illustrates_object",
                    "type": "array"
                },
                "publication_date": {
                    "format": "date-time",
                    "title": "publication_date",
                    "type": "string"
                },
                "title": {
                    "title": "title",
                    "type": "string"
                }
            },
            "required": [
                "title",
                "publication_date"
            ],
            "title": "Book",
            "type": "object"
        },
        "Illustration": {
            "additionalProperties": false,
            "properties": {
                "data": {
                    "format": "data-url",
                    "title": "data",
                    "type": "string"
                }
            },
            "required": [
                "data"
            ],
            "title": "Illustration",
            "type": "object"
        }
    }
}

We now have an additional illustrates property modelled as an array of #/definitions/Illustration, the later also added the the document as an additional definition entry.

Conclusion

This post illustrated how a basic (CRUD) HTTP API based on JSON Schema could be build for a CubicWeb application using cubicweb-jsonschema. We have seen a couple of details on JSON Schema generation and how it can be controlled. Feel free to comment and provide feedback on this feature set as well as open the discussion with more use cases.

Next time, we'll discuss how hypermedia controls can be added the HTTP API that cubicweb-jsonschema provides.

[1]	this choice is essentially driven by simplicity and conformance when the existing behavior to help migration of existing applications.

Monitor all the things! ... and early too!

Arthur Lutz — 2016-09-16T10:34-01:00

Following the "release often, release early" mantra, I thought it might be a good idea to apply it to monitoring on one of our client projects. So right from the demo stage where we deliver a new version every few weeks (and sometimes every few days), we setup some monitoring.

Monitoring performance

The project is an application built with the CubicWeb platform, with some ElasticSearch for indexing and searching. As with any complex stack, there are a great number of places where one could monitor performance metrics.

Here are a few things we have decided to monitor, and with what tools.

Monitoring CubicWeb

To monitor our running Python code, we have decided to use statsd, since it is already built into CubicWeb's core. Out of the box, you can configure a statsd server address in your all-in-one.conf configuration. That will send out some timing statistics about some core functions.

The statsd server (there a numerous implementations, we use a simple one : python-pystatsd) gets the raw metrics and outputs them to carbon which stores the time series data in whisper files (which can be swapped out for a different technology if need be).

If we are curious about a particular function or view that might be taking too long to generate or slow down the user experience, we can just add the @statsd_timeit decorator there. Done. It's monitored.

statsd monitoring is a fire-and-forget UDP type of monitoring, it should not have any impact on the performance of what you are monitoring.

Monitoring Apache

Simply enough we re-use the statsd approach by plugging in an apache module to time the HTTP responses sent back by apache. With nginx and varnish, this is also really easy.

One of the nice things about this part is that we can then get graphs of errors since we will differentiate OK 200 type codes from 500 type codes (HTTP codes).

Monitoring ElasticSearch

ElasticSearch comes with some metrics in GET /_stats endpoint, the same goes for individual nodes, individual indices and even at cluster level. Some popular tools can be installed through the ElasticSearch plugin system or with Kibana (plugin system there too).

We decided on a different approach that fitted well with our other tools (and demonstrates their flexibility!) : pull stats out of ElasticSearch with SaltStack, push them to Carbon, pull them out with Graphite and display them in Grafana (next to our other metrics).

On the SaltStack side, we wrote a two line execution module (elasticsearch.py)

import requests
def stats():
    return request.get('http://localhost:9200/_stats').json()

This gets shipped using the custom execution modules mechanism (_modules and saltutils.sync_modules), and is executed every minute (or less) in the salt scheduler. The resulting dictionary is fed to the carbon returner that is configured to talk to a carbon server somewhere nearby.

# salt demohost elasticsearch.stats
[snip]
  { "indextime_inmillis" : 30,
[snip]

Monitoring web metrics

To evaluate parts of the performance of a web page we can look at some metrics such as the number of assets the browser will need to download, the size of the assets (js, css, images, etc) and even things such as the number of subdomains used to deliver assets. You can take a look at such metrics in most developer tools available in the browser, but we want to graph this over time. A nice tool for this is sitespeed.io (written in javascript with phantomjs). Out of the box, it has a graphite outputter so we just have to add --graphiteHost FQDN. sitespeed.io even recommends using grafana to visualize the results and publishes some example dashboards that can be adapted to your needs.

The sitespeed.io command is configured and run by salt using pillars and its scheduler.

We will have to take a look at using their jenkins plugin with our jenkins continuous integration instance.

Monitoring crashes / errors / bugs

Applications will have bugs (in particular when released often to get a client to validate some design choices early). Level 0 is having your client calling you up saying the application has crashed. The next level is watching some log somewhere to see those errors pop up. The next level is centralised logs on which you can monitor the numerous pieces of your application (rsyslog over UDP helps here, graylog might be a good solution for visualisation).

When it starts getting useful and usable is when your bugs get reported with some rich context. That's when using sentry gets in. It's free software developed on github (although the website does not really show that) and it is written in python, so it was a good match for our culture. And it is pretty awesome too.

We plug sentry into our WSGI pipeline (thanks to cubicweb-pyramid) by installing and configuring the sentry cube : cubicweb-sentry. This will catch rich context bugs and provide us with vital information about what the user was doing when the crash occured.

This also helps sharing bug information within a team.

The sentry cube reports on errors being raised when using the web application, but can also catch some errors when running some maintenance or import commands (ccplugins in CubicWeb). In this particular case, a lot of importing is being done and Sentry can detect and help us triage the import errors with context on which files are failing.

Monitoring usage / client side

This part is a bit neglected for the moment. Client side we can use Javascript to monitor usage. Some basic metrics can come from piwik which is usually used for audience statistics. To get more precise statistics we've been told Boomerang has an interesting approach, enabling a closer look at how fast a page was displayed client side, how much time was spend on DNS, etc.

On the client side, we're also looking at two features of the Sentry project : the raven-js client which reports Javascript errors directly from the browser to the Sentry server, and the user feedback form which captures some context when something goes wrong or a user/client wants to report that something should be changed on a given page.

Load testing - coverage

To wrap up, we also often generate traffic to catch some bugs and performance metrics automatically :

wget --mirror $URL
linkchecker $URL
for $search_term in cat corpus; do wget URL/$search_term ; done
wapiti $URL --scope page
nikto $URL

Then watch the graphs and the errors in Sentry... Fix them. Restart.

Graphing it in Grafana

We've spend little time on the dashboard yet since we're concentrating on collecting the metrics for now. But here is a glimpse of the "work in progress" dashboard which combines various data sources and various metrics on the same screen and the same time scale.

Further plans

internal health checks, we're taking a look at python-hospital and healthz: Stop reverse engineering applications and start monitoring from the inside (Monitorama) (the idea is to distinguish between the app is running and the app is serving it's purpose), and pyramid_health
graph the number of Sentry errors and the number of types of errors: the sentry API should be able to give us this information. Feed it to Salt and Carbon.
setup some alerting : next versions of Grafana will be doing that, or with elastalert
setup "release version X" events in Graphite that are displayed in Grafana, maybe with some manual command or a postcreate command when using docker-compose up ?
make it easier for devs to have this kind of setup. Using this suite of tools in developement might sometimes be overkill, but can be useful.

Status of the CubicWeb python3 porting effort, February 2016

Julien Cristau — 2016-02-05T15:03-01:00

An effort to port CubicWeb to a dual python 2.6/2.7 and 3.3+ code base was started by Rémi Cardona in summer of 2014. The first task was to port all of CubicWeb's dependencies:

logilab-common 0.63
logilab-database 1.14
logilab-mtconverter 0.9
logilab-constraint 0.6
yams 0.40
rql 0.34

Once that was out of the way, we could start looking at CubicWeb itself. We first set out to make sure we used python3-compatible syntax in all source files, then started to go and make as much of the test suite as possible pass under both python2.7 and python3.4. As of the 3.22 release, we are almost there. The remaining pain points are:

cubicweb's setup.py hadn't been converted. This is fixed in the 3.23 branch as of https://hg.logilab.org/master/cubicweb/rev/0b59724cb3f2 (don't follow that link, the commit is huge)
the CubicWebServerTC test class uses twisted to start an http server thread, and twisted itself is not available for python3
the current method to serialize schema constraints into CWConstraint objects gives different results on python2 and python3, so it needs to be fixed (https://www.logilab.org/ticket/296748)
various questions around packaging and deployment: what happens to e.g. the cubicweb-common package installing into python2's site-packages directory? What does the ${prefix}/share/cubicweb directory become? How do cubes express their dependencies? Do we need a flag day? What does that mean for applications?

Moving forward on CubicWeb development

Sylvain Thenault — 2016-02-08T13:50-01:00

Some of us at Logilab made some kind of retrospective of the processes surrounding CubicWeb (core) development. One of the important point is that we decided that Logilab would not resume the bi-monthly meetings with other developers and users to discuss and update the roadmap.

Instead, we will do more Blog Driven Development (as started by Denis on this blog) to keep the community up to date with our internal effort. We will continue to use the mailing-list to discuss with everybody what we contribute to CubicWeb.

We will also try to organize a sprint in the coming months, but we are not able to commit on a date for now, because we are under a heavy load with customer work (which is a good thing for us and CubicWeb, of course).

Another topic was to set up a kind of working agreement for those developing on the core. For a start, we agreed on the following points:

we stop the two-steps review process
integrators commit to handling pending reviews under a 7 days time frame
an integrator is not allowed to integrate his or her own patches
incoming patches should not make the QA numbers go down, and should be py3k compatible.

QA numbers are still to be defined in a forthcoming internal sprint on continuous integration system. Current integrators are Julien, David, Denis, Florent and Sylvain. If you're interested, let us know on the mailing-list.

Using JSONAPI as a Web API format for CubicWeb

Denis Laxalde — 2016-02-08T13:03-01:00

Following the introduction post about rethinking the web user interface of CubicWeb, this article will address the topic of the Web API to exchange data between the client and the server. As mentioned earlier, this question is somehow central and deserves particular interest, and better early than late. Of the two candidate representations previously identified Hydra and JSON API, this article will focus on the later. Hopefully, this will give a better insight of the capabilities and limits of this specification and would help take a decision, though a similar experiment with another candidate would be good to have. Still in the process of blog driven development, this post has several open questions from which a discussion would hopefully emerge...

A glance at JSON API

JSON API is a specification for building APIs that use JSON as a data exchange format between clients and a server. The media type is application/vnd.api+json. It has a 1.0 version available from mid-2015. The format has interesting features such as the ability to build compound documents (i.e. response made of several, usually related, resources) or to specify filtering, sorting and pagination.

A document following the JSON API format basically represents resource objects, their attributes and relationships as well as some links also related to the data of primary concern.

Taking the example of a Ticket resource modeled after the tracker cube, we could have a JSON API document formatted as:

GET /ticket/987654
Accept: application/vnd.api+json

{
  "links": {
    "self": "https://www.cubicweb.org/ticket/987654"
  },
  "data": {
    "type": "ticket",
    "id": "987654",
    "attributes": {
      "title": "Let's use JSON API in CubicWeb"
      "description": "Well, let's try, at least...",
    },
    "relationships": {
      "concerns": {
        "links": {
          "self": "https://www.cubicweb.org/ticket/987654/relationships/concerns",
          "related": "https://www.cubicweb.org/ticket/987654/concerns"
        },
        "data": {"type": "project", "id": "1095"}
      },
      "done_in": {
        "links": {
          "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
          "related": "https://www.cubicweb.org/ticket/987654/done_in"
        },
        "data": {"type": "version", "id": "998877"}
      }
    }
  },
  "included": [{
    "type": "project",
    "id": "1095",
    "attributes": {
        "name": "CubicWeb"
    },
    "links": {
      "self": "https://www.cubicweb.org/project/cubicweb"
    }
  }]
}

In this JSON API document, top-level members are links, data and included. The later is here used to ship some resources (here a "project") related to the "primary data" (a "ticket") through the "concerns" relationship as denoted in the relationships object (more on this later).

While the decision of including or not these related resources along with the primary data is left to the API designer, JSON API also offers a specification to build queries for inclusion of related resources. For example:

GET /ticket/987654?include=done_in
Accept: application/vnd.api+json

would lead to a response including the full version resource along with the above content.

Enough for the JSON API overview. Next I'll present how various aspects of data fetching and modification can be achieved through the use of JSON API in the context of a CubicWeb application.

CRUD

CRUD of resources is handled in a fairly standard way in JSON API, relying of HTTP protocol semantics.

For instance, creating a ticket could be done as:

POST /ticket
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "data": {
    "type": "ticket",
    "attributes": {
      "title": "Let's use JSON API in CubicWeb"
      "description": "Well, let's try, at least...",
    },
    "relationships": {
      "concerns": {
        "data": { "type": "project", "id": "1095" }
      }
    }
  }
}

Then updating it (assuming we got its id from a response to the above request):

PATCH /ticket/987654
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "data": {
    "type": "ticket",
    "id": "987654",
    "attributes": {
      "description": "We'll succeed, for sure!",
    },
  }
}

Relationships

In JSON API, a relationship is in fact a first class resource as it is defined by a noun and an URI through a link object. In this respect, the client just receives a couple of links and can eventually operate on them using the proper HTTP verb. Fetching or updating relationships is done using the special <resource url>/relationships/<relation type> endpoint (self member of relationships items in the first example). Quite naturally, the specification relies on GET verb for fetching targets, PATCH for (re)setting a relation (i.e. replacing its targets), POST for adding targets and DELETE to drop them.

GET /ticket/987654/relationships/concerns
Accept: application/vnd.api+json

{
  "data": {
    "type": "project",
    "id": "1095"
  }
}

PATCH /ticket/987654/relationships/done_in
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "data": {
    "type": "version",
    "id": "998877"
  }
}

The body of request and response of this <resource url>/relationships/<relation type> endpoint consists of so-called resource identifier objects which are lightweight representation of resources usually only containing information about their "type" and "id" (enough to uniquely identify them).

Related resources

Remember the related member appearing in relationships links in the first example?

  [ ... ]
  "done_in": {
    "links": {
      "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
      "related": "https://www.cubicweb.org/ticket/987654/done_in"
    },
    "data": {"type": "version", "id": "998877"}
  }
  [ ... ]

While this is not a mandatory part of the specification, it has an interesting usage for fetching relationship targets. In contrast with the .../relationships/... endpoint, this one is expected to return plain resource objects (which attributes and relationships information in particular).

GET /ticket/987654/done_in
Accept: application/vnd.api+json

{
  "links": {
    "self": "https://www.cubicweb.org/998877"
  },
  "data": {
    "type": "version",
    "id": "998877",
    "attributes": {
        "number": 4.2
    },
    "relationships": {
      "version_of": {
        "self": "https://www.cubicweb.org/998877/relationships/version_of",
        "data": { "type": "project", "id": "1095" }
      }
    }
  },
  "included": [{
    "type": "project",
    "id": "1095",
    "attributes": {
        "name": "CubicWeb"
    },
    "links": {
      "self": "https://www.cubicweb.org/project/cubicweb"
    }
  }]
}

Meta information

The JSON API specification allows to include non-standard information using a so-called meta object. This can be found in various place of the document (top-level, resource objects or relationships object). Usages of this field is completely free (and optional). For instance, we could use this field to store the workflow state of a ticket:

{
  "data": {
    "type": "ticket",
    "id": "987654",
    "attributes": {
      "title": "Let's use JSON API in CubicWeb"
    },
    "meta": { "state": "open" }
}

Permissions

Permissions are part of metadata to be exchanged during request/response cycles. As such, the best place to convey this information is probably within the headers. According to JSON API's FAQ, this is also the recommended way for a resource to advertise on supported actions.

So for instance, response to a GET request could include Allow headers, indicating which request methods are allowed on the primary resource requested:

GET /ticket/987654
Allow: GET, PATCH, DELETE

An HEAD request could also be used for querying allowed actions on links (such as relationships):

HEAD /ticket/987654/relationships/comments
Allow: POST

This approach has the advantage of being standard HTTP, no particular knowledge of the permissions model is required and the response body is not cluttered with these metadata.

Another possibility would be to rely use the meta member of JSON API data.

{
  "data": {
    "type": "ticket",
    "id": "987654",
    "attributes": {
      "title": "Let's use JSON API in CubicWeb"
    },
    "meta": {
      "permissions": ["read", "update"]
    }
  }
}

Clearly, this would minimize the amount client/server requests.

More Hypermedia controls

With the example implementation described above, it appears already possible to manipulate several aspects of the entity-relationship database following a CubicWeb schema: resources fetching, CRUD operations on entities, set/delete operations on relationships. All these "standard" operations are discoverable by the client simply because they are baked into the JSON API format: for instance, adding a target to some relationship is possible by POSTing to the corresponding relationship resource something that conforms to the schema.

So, implicitly, this already gives us a fairly good level of Hypermedia control so that we're not so far from having a mature REST architecture according to the Richardson Maturity Model. But beyond these "standard" discoverable actions, the JSON API specification does not address yet Hypermedia controls in a generic manner (see this interesting discussion about extending the specification for this purpose).

So the question is: would we want more? Or, in other words, do we need to define "actions" which would not map directly to a concept in the application model?

In the case of a CubicWeb application, the most obvious example (that I could think of) of where such an "action" would be needed is workflow state handling. Roughly, workflows in CubicWeb are modeled through two entity types State and TrInfo (for "transition information"), the former being handled through the latter, and a relationship in_state between the workflowable entity type at stake and its current State. It does not appear so clearly how would one model this in terms of HTTP resource. (Arguably we wouldn't want to expose the complexity of Workflow/TrInfo/State data model to the client, nor can we simply expose this in_state relationship, as a client would not be able to simply change the state of a entity by updating the relation). So what would be a custom "action" to handle the state of a workflowable resource? Back in our tracker example, how would we advertise to the client the possibility to perform "open"/"close"/"reject" actions on a ticket resource? Open question...

Request for comments

In this post, I tried to give an overview of a possible usage of JSON API to build a Web API for CubicWeb. Several aspects were discussed from simple CRUD operations, to relationships handling or non-standard actions. In many cases, there are open questions for which I'd love to receive feedback from the community. Recalling that this topic is a central part of the experiment towards building a client-side user interface to CubicWeb, the more discussion it gets, the better!

For those wanting to try and play themselves with the experiments, have a look at the code. This is a work-in-progress/experimental implementation, relying on Pyramid for content negotiation and route traversals.

What's next? Maybe an alternative experiment relying on Hydra? Or an orthogonal one playing with the schema client-side?

Happy New Year CubicWeb !

Nicolas Chauvat — 2016-01-25T13:30-01:00

This CubicWeb blog that has been asleep for some months, whereas the development was active. Let me try to summarize the recent progress.

CubicWeb 3.21

CubicWeb 3.21 was published in July 2015. The announce was sent to the mailing list and changes were listed in the documentation.

The main goal of this release was to reduce the technical debt. The code was improved, but the changes were not directly visible to users.

CubicWeb 3.22

CubicWeb 3.22 was published in January 2016. A mail was sent to the mailing list and the documentation was updated with the list of changes.

The main achievements of this release were the inclusion of a new procedure to massively import data when using a Postgresql backend, improvements of migrations and customization of generic JSON exports.

Roadmap and bi-monthly meetings

After the last-minute cancellation of the may 2015 roadmap meeting, we failed to reschedule in june, the summer arrived, then the busy-busy end of the year... and voilà, we are in 2016.

During that time, Logilab has been working on massive data import, full-js user interfaces exchanging JSON with the CubicWeb back-end, 3D in the browser, switching CubicWeb to Python3, moving its own apps to Bootstrap, using CubicWeb-Pyramid in production and improving management/supervision, etc. We will be more than happy to discuss this with the rest of the (small but strong) CubicWeb community.

So let's wish a happy new year to everyone and meet again in March for a new roadmap session !

Towards building a JavaScript user interface to CubicWeb

Denis Laxalde — 2016-01-27T21:15-01:00

This post is an introduction of a series of articles dealing with an on-going experiment on building a JavaScript user interface to CubicWeb, to ultimately replace the web component of the framework. The idea of this series is to present the main topics of the experiment, with open questions in order to eventually engage the community as much as possible. The other side of this is to experiment a blog driven development process, so getting feedback is the very point of it!

As of today, three main topics have been identified:

the Web API to let the client and server communicate,
the issue of representing the application schema client-side, and,
the construction of components of the web interface (client-side).

As part of the first topic, we'll probably rely on another experimental work about REST-fulness undertaken recently in pyramid-cubicweb (see this head for source code). Then, it appears quite clearly that we'll need sooner or later a representation of data on the client-side and that, quite obviously, the underlying format would be JSON. Apart from exchanging of entities (database) information, we already anticipate on the need for the HATEOAS part of REST. We already took some time to look at the existing possibilities. At a first glance, it seems that hydra is the most promising in term of capabilities. It's also built using semantic web technologies which definitely grants bonus point for CubicWeb. On the other hand, it seems a bit isolated and very experimental, while JSON API follows a more pragmatic approach (describe itself as an anti-bikeshedding tool) and appears to have more traction from various people. For this reason, we choose it for our first draft, but this topic seems so central in a new UI, and hard to hide as an implementation detail; that it definitely deserves more discussion. Other candidates could be Siren, HAL or Uber.

Concerning the schema, it seems that there is consensus around JSON-Schema so we'll certainly give it a try.

Finally, while there is nothing certain as of today we'll probably start on building components of the web interface using React, which is also getting quite popular these days. Beyond that choice, the first practical task in this topic will concern the primary view system. This task being neither too simple nor too complicated will hopefully result in a clearer overview of what the project will imply. Then, the question of edition will come up at some point. In this respect, perhaps it'll be a good time to put the UX question at a central place, in order to avoid design issues that we had in the past.

Feedback welcome!