A Technology Stack on Top of a (Neo4j) Graph Database

Putting it All Together : a Technology Stack on top of a (Neo4j) Graph Database

(UPDATED June 2024) For many practical use cases, one needs a full data-management solution, not just a database.

So, armed with the Schema Layer discussed in the previous part, the next natural step is to add a web API and possibly a User Interface.

This article is part of a growing, ongoing series on Graph Databases and Neo4j

The Web API Layer / Data Manager

The Web API Layer ("Data Manager") is in some ways the most straightforward layer - because the "heavy lifting" is done by the Schema Layer.

I've been involved in projects that utilized an inadequate Schema Layer - and in those situations the API Layer ends up taking on an immense amount of responsibility that don't logically belong there; the end result being a lot of difficult, error-prone and non-modular development that feels like "pulling teeth"!

By contrast, with a well-developed Schema Layer that provides a wide range of services, it becomes a relative breeze to build an API Layer. The API Layer can then concentrate on roles that logically belong there, such as receiving JSON commands, parsing them, and invoking the appropriate methods of the Schema Layer and of the specialized libraries, which do most of the work.

With such a well-defined modular role, the development focus of the API Layer can be to provide a set of useful, convenient, and intuitive request-handling commands, typically for the UI and/or to provide a web endpoint.

The BrainAnnex.org project uses the Flask platform to provide a number of typical operations in this layer - such as to add or edit nodes in the database - and it's fairly easy to add additional operations... or duplicate this layer to meet one's needs.

The API Layer can also take on responsibilities such as:

authentication
implementing a master-slave architecture that doesn't require direct Neo4j database support (which would necessitate the licensed version of Neo4j.)

The Edge Between the Back End and the Front End

Let's consider a more detailed diagram of the architecture used in the Brain Annex project:

The above technology stack is explained in this short video.

Note that Flask is being used for two separate purposes in this architecture:

1) to implement a web API

2) to serve dynamic web pages (with embedded Vue.js - or in principle REACT, etc - for high interactivity)

The reason I favor this combined approach is that it lends itself very well to the philosophy of gradual incremental changes : at an early stage, Flask can serve pages with little interactivity (such as reports, or just simple controls), to get things started. Over time, increased levels of interactivity can get grafted in, leading to a gradual handover of responsibilities from Flask to Vue (or other framework).

In the very early version of the web-app page, Flask has full responsibility over it - which won't win awards for graceful interactivity, but is infinitely better than not having anything!

In later versions, as development on that page progresses, Flask does little other that pass over the data to the front-end framework (embedded in the page and served by Flask); at page load, the front-end framework takes over most or all of the presentation and interactivity.

Of course, the front-end framework can also query the web API for any data not passed to it by Flask at page-creation time.

For submitting data to the server, such as forms, Flask can do in a clumsy "web 1.0" way (with page reloads) - which is a way to tide over until, upon further development, the front end takes over those tasks, and can do them with elegant interactivity, with no page reloads.

An alternate architecture, probably a more common approach, is to have standalone pages (NOT served by Flask), purely managed by the front-end framework... and only interacting with the backend thru web API calls. That's perfectly fine - but in that case you're starting your pages with zero, rather than enjoying a running start with quick-'n-dirty pages easily schlepped together by Flask. The Brain Annex project favors incremental approaches - but it's modular, and you can opt to utilize its layers thru the web API, and then do your own front end as it suits you.

The User Interface Layer

The API Layer takes care of the data management; so that the UI can focus on the front-end user experience.

As far as I'm concerned, excellence in the UI is truly "the final frontier"!

That's the layer that ultimately makes the fundamental difference for the user. Most users won't care if the system has "the engine of a Ferrari", if the vehicle looks and feels like a beaten-up junk car!

All the other layers are in place, and - while there's plenty of room for improvement (and for more documentation) - the foundations are fairly solid and extensive, and as of June 2024 the Brain Project layers below the UI are a hair away from leaving a late-Beta stage (and the NeoAccess library left the Beta stage long ago.) The UI is a different story; it's the cutting edge of development - and a lot of possibilities exist to explore.

The BrainAnnex.org project uses the Vue.js platform and the Cytoscape.js library to provide a dual modality for displaying/editing data: tabular form and graph form ("balls and edges".)

Also, BrainAnnex makes use of a Class-based UI, whereby some of the data Classes (as defined in the Schema layer) are associated with ad-hoc software ("plugins") to display/edit data of that class in a particular manner. Historically, the early versions of BrainAnnex utilized this approach to implement a multimedia content management system - but now that's just one use case.

The UI Layer is a very active area of development. Just like for the API Layer, one might opt to supplement, based on one's needs, the existing layer provided by BrainAnnex - or replace it with a custom one.

Things start getting especially exciting and powerful when the UI is aware of the different data types (for example, as specified in the Schema layer), and has the capability to personalize the display and editing mode of data records based on their types ("classes") - perhaps with plugins to provide modularity and easy expansion.

Well, that's exactly what the new version (5) of the open-source project Brain Annex does!

One use case of such a system is to be a multimedia content management system, which is what the old ("vintage") versions of Brain Annex was, before switching to being Neo4j-based, and morphing into a far more general system, a transition that started in late 2020 (project's history.)

Armed with an API, and possibly a UI, one can for example create a standalone web app, or a control panel for an existing website or web app. For example, the diagram at the very top of this page is how Brain Annex does it.

For more information about this layer, please see this short video.

Special-Purpose Modules

Now that we have a whole technology stack, how about looking into special-purpose modules? We'll do that across articles on a variety of special topics, such as a discussion (and available open-source implementation) of full-text search!

This article is part of a growing, ongoing series on Graph Databases and Neo4j

Comments

Discussing Neuroscience with ChatGPT

UPDATED Apr. 2023 - I'm excited by ChatGPT 's possibilities in terms of facilitating advanced learning . For example, I got enlightening answers to questions that I had confronted when I first studied neuroscience. The examples below are taken from a very recent session I had with ChatGPT (mid Jan. 2023.) Source: https://neurosciencestuff.tumblr.com In case you're not familiar with ChatGPT, it's a very sophisticated "chatbot" - though, if you call it that way, it'll correct you! 'I am not a "chatbot", I am a language model, a sophisticated type of AI algorithm trained on vast amounts of text data to generate human-like text'. For a high-level explanation of how ChatGPT actually works - which also gives immense insight into its weaknesses, there's an excellent late Jan. 2023 talk by Stephen Wolfram, the brilliant author of the Mathematica software and of Wolfram Alpha , a product that could be combined with ChatGPT to imp...

Using Schema in Graph Databases such as Neo4j

UPDATED Feb. 2024 - Graph databases have an easygoing laissez-faire attitude: "express yourself (almost) however you want"... By contrast, relational databases come across with an attitude like a micro-manager: "my way or the highway"... Is there a way to take the best of both worlds and distance oneself from their respective excesses, as best suited for one's needs? A way to marry the flexibility of Graph Databases and the discipline of Relational Databases? This article is part 5 of a growing, ongoing series on Graph Databases and Neo4j Let's Get Concrete Consider a simple scenario with scientific data such as the Sample, Experiment, Study, Run Result , where Samples are used in Experiments, and where Experiments are part of Studies and produce Run Results. That’s all very easy and intuitive to represent and store in a Labeled Graph Database such as Neo4j . For example, a rough draft might go like this: The “labels” (b...

Graph Databases (Neo4j) - a revolution in modeling the real world!

UPDATED Oct. 2023 - I was "married" to Relational Databases for many years... and it was a good "relationship" full of love and productivity - but SOMETHING WAS MISSING! Let me backtrack. In college, I got a hint of the "pre-relational database" days... Mercifully, that was largely before my time, but - primarily through a class - I got a taste of what the world was like before relational databases. It's an understatement to say: YUCK! Gratitude for the power and convenience of Relational Databases and SQL - and relief at having narrowly averted life before it! - made me an instant mega-fan of that technology. And for many years I held various jobs that, directly or indirectly, made use of MySQL and other relational databases - whether as a Database Administrator, Full-Stack Developer, Data Scientist, CTO or various other roles. UPDATE: This article is now part 1 of a growing, ongoing series on Graph Databases and Neo4j But ther...

What are Graph Databases - and Why Should I Care?? : "Graph Databases for Poets"

This is a very gentle introduction to the subject. The subtitle is inspired by university courses such as "Physics for Poets"! (if you're technically inclined, there's an alternate article for you.) It has been said that "The language of physics (or of God) is math". On a similar note, it could be said that: The language of the biological world - or of any subject or endeavor involving complexity - is networks ('meshes') What is a network? Think of it as the familiar 'friends of friends' diagram from social media. Everywhere one turns in biology, there's a network – at the cellular level, tissue level, organ level, ecosystem level. The weather and other earth systems are networks. Human societal organization is a network. Electrical circuits, the Internet, our own brains... Networks are everywhere! What can we do with networks, to better understand the world around us, or to create something that we need? Broadly ...

Anti-Aging Research: Science, not Hype

Last updated May 2023 Q: "How is aging a disease?" A: It's a dynamic system that veers away from its homeostasis (normal equilibrium point): hence a form of slow-progressing illness. Labeling it as 'natural' is a surrender to our traditional state of ignorance and powerlessness, which fortunately is beginning to be changed! Aging is "normal" only from the point of view of the "selfish gene", for whom the body is a disposable carrier. Individuals organisms - for whom self-preservation has a different meaning than for genes - have received scant help from evolution... with rare exceptions such as the T. dohrnii jellyfish (which I discuss here )... but now the time has finally arrived for our rational design to remedy some of the cellular flaws that evolution never bothered to correct! The above is my standard answer to an oft-asked question. The science of aging is by all evidence very misunderstood by the general public. Hype,...

Using Neo4j with Python : the Open-Source Library "NeoAccess"

So, you want to build a python app or Jupyter notebook to utilize Neo4j, but aren't too keen on coding a lot of string manipulation to programmatic create ad-hoc Cypher queries? You're in the right place: the NeoAccess library can do take care of all that, sparing you from lengthy, error-prone development that requires substantial graph-database and software-development expertise! This article is part 4 of a growing, ongoing series on Graph Databases and Neo4j "NeoAccess" is the bottom layer of the technology stack provided by the BrainAnnex open-source project . All layers are very modular, and the NeoAccess library may also be used by itself , entirely separately from the rest of the technology stack. (A diagram of the full stack is shown later in this article.) NeoAccess interacts with the Neo4j Python driver , which is provided by the Neo4j company, to access the database from Python; the API to access that driver is very p...

Visualization of Graph Databases Using Cytoscape.js

(UPDATED APR. 2024) I have ample evidence from multiple sources that there are strong unmet needs in the area of visualization of graph databases. And whenever there's a vacuum, vendors circle like vultures - with incomplete, non-customizable, and at times ridiculously expensive, closed-box proprietary solutions. Fortunately, coming to the rescue is the awesome open-source cytoscape.js library , an offshoot of the "Cytoscape" project of the Institute for Systems Biology , a project with a long history that goes back to 2002. One can do amazing custom solutions, relatively easily, when one combines this Cytoscape library with: 1) a front-end framework such as Vue.js 2) backend libraries (for example in python) to prepare and serve the data For example, a while back I created a visualizer for networks of chemical reactions, for another open-source project I lead ( life123.science ) This visualizer will look and feel generally ...

Neo4j Sandbox Tutorial : try Neo4j and learn Cypher - free and easy!

So, you have an itch to test-drive Neo4j and its Cypher query language. Maybe you want to learn it, or evaluate it, or introduce colleagues/clients to it. And you wish for: fast, simple and free! Well, good news: the Neo4j company kindly provides a free, short-term hosted solution called "the Neo4j sandbox" . Extremely easy to set up and use! This article is part 2 of a growing, ongoing series on Graph Databases and Neo4j Register (free) for the Neo4j "Sandbox" Go to sandbox.neo4j.com , and register with a working email and a password. That's it! Note that this same email/password will also let you into the Neo4j Community Forums and Support ; the same login for all: very convenient! Launch your instance - blank or pre-populated After registering, go to sandbox.neo4j.com , and follow the steps in the diagram below (the choices might differ, but the "Blank Sandbox" should always be there): Too good to be true? Is there...

Neo4j & Cypher Tutorial : Getting Started with a Graph Database and its Query Language

You have a general idea of what Graph Databases - and Neo4j in particular - are... But how to get started? Read on! This article is part 3 of a growing, ongoing series on Graph Databases and Neo4j If you're new to graph databases, please check out part 1 for an intro and motivation about them. There, we discussed an example about an extremely simple database involving actors, movies and directors... and saw how easy the Cypher query language makes it to answer questions such as "which directors have worked with Tom Hanks in 2016" - questions that, when done with relational databases and SQL, turn into a monster of a query and an overly-complicated data model involving a whopping 5 tables! In this tutorial, we will actually carry out that query - and get acquainted with Cypher and the Neo4j browser interface in the process. This is the dataset we'll be constructing: Get the database in place If you don't already have a datab...

Full-Text Search with the Neo4j Graph Database

(UPDATED May 2024) Now that we have discussed a full technology stack based on Neo4j (or other graph databases), and that we a design and implementation available from the open-source project BrainAnnex.org , what next? What shall we build on top? Well, how about Full-Text Search ? This article is part of a growing, ongoing series on Graph Databases and Neo4j Full-Text Searching/Indexing The Brain Annex open-source project includes an implementation of a design that uses the convenient services of its Schema Layer , to provide indexing of word-based documents using Neo4j. The python class FullTextIndexing ( source code ) provides the necessary methods, and it can parse both plain-text and HTML documents (for example, used in "formatted notes"); parsing of PDF files and other formats will be added at a later date. No grammatical analysis ( stemming or lemmatizing ) is done on the text. However, a long list of common word ("stop ...

Julian's Polymath Explorations

Search This Blog