Skip to main content

Using Neo4j with Python : the Open-Source Library "NeoAccess"

So, you want to build a python app or Jupyter notebook to utilize Neo4j, but aren't too keen on coding a lot of string manipulation to programmatic create ad-hoc Cypher queries?   You're in the right place: the NeoAccess library can do take care of all that, sparing you from lengthy, error-prone development that requires substantial graph-database and software-development expertise!


This is part 4 of a 7-part series on Graph Databases and Neo4j. 

part 1 : Graph Databases (Neo4j) - a revolution in modeling the real world!

part 2 : Neo4j Sandbox Tutorial : try Neo4j and learn Cypher – free and easy

part 3 : Neo4j & Cypher Tutorial : Getting Started with a Graph Database and its Query Language 

part 4 : Using Neo4j with Python : the Open-Source Library "NeoAccess"

part 5 : Using Schema in Graph Databases such as Neo4j

part 6 : Putting it All Together - a Technology Stack on Top of a Graph Database

part 7 : (SPECIAL TOPIC) Full-Text Search with the Neo4j Graph Database

"NeoAccess" is the bottom layer of the technology stack provided by the BrainAnnex open-source project.  All layers are very modular, and the NeoAccess library may also be used by itself, entirely separately from the rest of the technology stack.  (A diagram of the full stack is shown later in this article.)

NeoAccess interacts with the Neo4j Python driver, which is provided by the Neo4j company, to access the database from Python; the API to access that driver is very powerful, but complex - and does not provide higher-level services.

The Handover from Graph Database Specialists to Data Scientists & General Python Programmers

One could do a lot directly with the Cypher query language...  And there are times when working directly with Cypher is necessary or convenient - just like when one may want to work interactively with SQL in a relational database. 

But what if you need to do something programmatically?

Or what if you don't want to deal with writing Cypher queries, or if someone working on the project may not be an expert in Cypher?

Well, that's where the NeoAccess library comes in to save the day!

In some ways, the NeoAccess library can be thought of as a handover from data engineering that requires expertise in Cypher, passing the baton to data scientists and software engineers who may have only limited understanding of what graph databases are, and limited or no knowledge of Cypher.

The goal of NeoAccess is to make it easy and convenient for a Python programmer or Data Scientist to manage a lot of the typical database operations - without having to deal with Cypher.

If you're accessing Neo4j programmatically, and you wanted to create functions to provide typical database operations (such as create a new node, or link 2 existing nodes), you'd be faced with a lot of tedious - and very error prone - string manipulations to create the Cypher queries in your Python scripts.  Not to mention having to deal with pesky special cases (such as, can one have blank spaces in a node label?  What if I later need to have more than 1 label per node?)

Those aggravations are something best avoided - and left to the NeoAccess library, which has grappled with these issues for a long time!  And it has been unit-tested at great length...

Example: Create a New Node

By using the NeoAccess library, all you have to do is: create a connection database, and then run the method "create_node" and pass arguments to it.

That's all!  (Compare to the Cypher query-language equivalent, below...)

And if you want to have a second "label" for the node, just change the label string into a list - and you don't have to worry about it!  Similarly, if you have blank spaces in any of the labels, you can just simply do it and not worry about it.

The NeoAccess library will worry about the Cypher syntax needed to deal with multiple labels, or needed to deal with blank spaces and labels.

No Deep Graph-Database Expertise Needed for Many Typical Operations

Some of the beauty of using a library like NeoAccess is that all you need is a relatively modest knowledge of the essentials of graph databases; something that you can really explain very quickly, with a diagram in the style of the familiar "friends-of-friends" (as done, for example, in Part 1 of this series of articles.)

As long as someone knows the basics... knows about nodes representing records with fields in it... and knows that nodes can be connected by relationships (aka links) that have a name and can also have attributes...  and, finally, that each node can have one or more labels (which are reminiscent of table names in relation databases)...

Well, that's the fundamental knowledge - and just about all that one needs to know - to do quite a bit using the NeoAccess library!

Conversely, people who are experts in Cypher, can easily submit any arbitrary query to NeoAccess - and get the results (including errors and status of the operations) in a variety of convenient Python data structures.

Block Diagram of NeoAccess

Repeated from the top of this article, here's a block diagram of what the the NeoAccess library consists of:

Shown at the center, there's a Foundational Layer, and a set of Cypher utilities.

The foundational layer takes care of operations like connecting to the database, running generic Cypher queries, and receiving complex data structures (and status/error info) from the database.

The Cypher utilities take care of a lot of the very specific syntactic operations to create correct Cypher queries.  

Armed with this foundation layer, one can easily run any Cypher query.

But in practice, most of the time, all that one needs to do are a small set of very typical operations.  For example: create notes, delete notes, modify fields, add/delete relationships, restructure relationships, follow links.  As well as operations involving labels, indexes and constraints.

NeoAccess allows the execution of those typical scenarios with simple, streamlined function calls.

It also provides a group operations about imports and exports.  For example, dealing with files or Pandas.   Oftentimes, the import/export operations are done at a higher layer, such as the schema layer (see next section), not at the NeoAccess layer.  So this is something of a separate group - and might end up getting split off in the future.

Who uses NeoAccess, and how does one use it?

The biggest user of the NeoAccess library is the NeoSchema library, the layer above it in the technology stack of the BrainAnnex open-source project:

And the layers above the NeoSchema library could in principle also use NeoAccess directly - although it's probably not a good idea; that's why I'm using dashed arrows there.

But anyone can use the NeoAccess library directly, if they so wish: you can use it with a Python script, or you can use it with a Jupyter notebook.  The remainder of the technology stack shown above is NOT required; if it doesn't suit your needs, you can simply opt to use the NeoAccess library directly.  

In situations where you don't really care about the schema, maybe because you're doing exploratory analysis or a demo, that's a use case of using the NeoAccess library directly, and by itself.  (In fact, it is independently released, and all you need to do to use it, is to install that one library from the standard python library repository.)

But in most situations, it may be better to go through the NeoSchema layer (the 2nd layer from the bottom, in the above diagram.)  However, be aware that the NeoSchema layer is currently (as of June 2023) in late Beta, whereas the NeoAccess library has been stable - and out of beta - for quite some time.

Brief History and Current Status

Back in 2020 I created, and released to open source, a library called Neo4jLiaison, now obsolete, which was a forerunner of what has become the NeoAccess library.

In the following year, 2021, I was working at pharmaceutical giant GSK.  At that time, I combined my library with another library that someone else in the company had created.  

Our two libraries conveniently happened to be fairly complimentary.  So, we merged them, and then expanded the combined library quite substantially - and added a lot of unit testing.

After a good part of a year of development, GSK pharmaceuticals very graciously released it as open source at the end of 2021, under the name of the NeoInterface library - after requiring a VERY EXTENSIVE amount of unit testing.

At that point I forked the open-source library, renamed it NeoAccess - and expanded it a fair bit throughout 2022 and 2023, using it as the new foundation of the overhauled BrainAnnex open-source project (which dates back to 2015, and was originally built on top of a relational database), as well as at 2 other use cases at jobs I held during that time. 

In 2023, the NeoAccess library started being distributed  independently from the rest of the BrainAnnex technology stack.

How to Use NeoAccess

IMPORTANT: starting Apr. 2023, the NeoAccess library is distributed independently , as a python package named "neoaccess" in the standard PyPI distribution.  One reason for the separate distribution, besides modularity, is that the NeoAccess library is stable and well-tested, while the upper layers of the BrainAnnex technology stack are in a late Beta stage.

You do NOT need to install the full BrainAnnex app if you only need the NeoAccess library!

To use the NeoAccess library:  pip install it from the standard PyPI distribution  (e.g., place neoaccess in your "requirements.txt" file, or use your IDE to install the latest "neoaccess" library).  Then, in your code, include:

from neoaccess import NeoAccess

At that point, you can issue commands such as:

host="you_host_info"    # EXAMPLES:  bolt://1.2.3.4  OR  neo4j://localhost

db = NeoAccess(host=host,  credentials=("neo4j", your_password_info))

neo_car = db.create_node("Car", {'color': 'white', 'make': 'Toyota'})

neo_person = db.create_node("Person", {'name': 'Julian'})

number_links_added = db.add_links(neo_car, neo_person, rel_name="OWNED_BY")

Of course, you will also need an installed Neo4j database - or simply register for the free "sandbox" short-term hosted solution described in a previous article.   (Be aware that the "sandbox" is for the very new version 5 of the Neo4j database, while the NeoAccess library is based on the well-established 4.x versions of the database; however, most basic functionality seems to hold up.)

Tutorials

They are in the form of annotated and illustrated Jupyterlab notebooks, which come as part of the Brain Annex repository:

Tutorial 1

Tutorial 2

The above link are just VIEWS of notebooks.  If you want to actually run them from your machine, follow the instructions in the previous section.

Further Directions

Programmer's Guide

List of functions in the NeoAccess library

The NeoAccess library repository

Versions & changelog (combined with the whole BrainAnnex technology stack)

This is part 4 of a 7-part series on Graph Databases and Neo4j. 

part 1 : Graph Databases (Neo4j) - a revolution in modeling the real world!

part 2 : Neo4j Sandbox Tutorial : try Neo4j and learn Cypher – free and easy

part 3 : Neo4j & Cypher Tutorial : Getting Started with a Graph Database and its Query Language 

part 4 : Using Neo4j with Python : the Open-Source Library "NeoAccess"

part 5 : Using Schema in Graph Databases such as Neo4j

part 6 : Putting it All Together - a Technology Stack on Top of a Graph Database

part 7 : (SPECIAL TOPIC) Full-Text Search with the Neo4j Graph Database

Comments

Popular posts from this blog

Discussing Neuroscience with ChatGPT

UPDATED Apr. 2023 - I'm excited by ChatGPT 's possibilities in terms of facilitating advanced learning .  For example, I got enlightening answers to questions that I had confronted when I first studied neuroscience.  The examples below are taken from a very recent session I had with ChatGPT (mid Jan. 2023.) Source: https://neurosciencestuff.tumblr.com In case you're not familiar with ChatGPT, it's a very sophisticated "chatbot" - though, if you call it that way, it'll correct you!  'I am not a "chatbot", I am a language model, a sophisticated type of AI algorithm trained on vast amounts of text data to generate human-like text'. For a high-level explanation of how ChatGPT actually works - which also gives immense insight into its weaknesses, there's an excellent late Jan. 2023 talk by Stephen Wolfram, the brilliant author of the Mathematica software and of Wolfram Alpha , a product that could be combined with ChatGPT to imp

Neo4j & Cypher Tutorial : Getting Started with a Graph Database and its Query Language

You have a general idea of what Graph Databases - and Neo4j in particular - are...  But how to get started?  Read on! This is part 3 of a 7-part series on Graph Databases and Neo4j.   part 1 : Graph Databases (Neo4j) - a revolution in modeling the real world! part 2 : Neo4j Sandbox Tutorial : try Neo4j and learn Cypher the free and easy way part 3 : Neo4j & Cypher Tutorial : Getting Started with a Graph Database and its Query Language  part 4 : Using Neo4j with Python : the Open-Source Library NeoAccess part 5 : Using Schema in Graph Databases such as Neo4j part 6  : Putting it All Together - a Technology Stack on Top of a Graph Database part 7  : (SPECIAL TOPIC) Full-Text Search with the Neo4j Graph Database If you're new to graph databases, please check out part 1 for an intro and motivation about them.  There, we discussed an example about an extremely simple database involving actors, movies and directors...  and saw how easy the Cypher query lan

Graph Databases (Neo4j) - a revolution in modeling the real world!

UPDATED July 2023 - I was "married" to Relational Databases for many years... and it was a good "relationship" full of love and productivity - but SOMETHING WAS MISSING! Let me backtrack.   In college, I got a hint of the "pre-relational database" days...  Mercifully, that was largely before my time, but  - primarily through a class - I got a taste of what the world was like before relational databases.  It's an understatement to say: YUCK! Gratitude for the power and convenience of Relational Databases and SQL - and relief at having narrowly averted life before it! - made me an instant mega-fan of that technology.  And for many years I held various jobs that, directly or indirectly, made use of MySQL and other relational databases - whether as a Database Administrator, Full-Stack Developer, Data Scientist, CTO or various other roles. UPDATE: This article is now part of a series... This is part 1 of a 7-part series on Graph Databases and Neo4j.  

To Build or Not to Build One’s Own Desktop Computer?

“ VALENTINA ” [UPDATED JUNE 2021] - Whether you're a hobbyist, or someone who just needs a good desktop computer, or an IT professional who wants a wider breath of knowledge, or a gamer who needs a performant machine, you might have contemplated at some point whether to build your own desktop computer. If you're a hobbyist, I think it's a great project.  If you're an IT professional - especially a "coder" - I urge you to do it: in my opinion, a full-fledged Computer Scientist absolutely needs breath, ranging from the likes of Shannon's Information Theory and the Halting Problem - all the way down to how transistors work. And what about someone who just needs a good desktop computer?  A big maybe on that - but perhaps this blog entry will either help you, or scare you off for your own good! To build, or not to build, that is the question: Whether 'tis nobler in the mind to suffer The slings and arrows of OEM's cutting corners and limit

Using Schema in Graph Databases such as Neo4j

UPDATED Aug. 2023 - Graph databases have an easygoing laissez-faire attitude: "express yourself (almost) however you want"... By contrast, relational databases come across with an attitude along the lines of a micro-manager:  "my way or the highway"... Is there a way to take the best of both worlds and distance oneself from their respective excesses, as best suited for one's needs? This is part 5 of a 7-part series on Graph Databases and Neo4j.   part 1 : Graph Databases (Neo4j) - a revolution in modeling the real world! part 2 : Neo4j Sandbox Tutorial : try Neo4j and learn Cypher the free and easy way part 3 : Neo4j & Cypher Tutorial : Getting Started with a Graph Database and its Query Language part 4 : Using Neo4j with Python : the Open-Source Library NeoAccess part 5 : Using Schema in Graph Databases such as Neo4j part 6  : Putting it All Together - a Technology Stack on Top of a Graph Database part 7  : (SPECIAL TOPIC) F

Full-Text Search with the Neo4j Graph Database

(UPDATED Sep. 2023)   In part 5 ( Using Schema in Graph Databases ) we discussed the concept of a Schema Layer, and a design and implementation available from the open-source project BrainAnnex.org Now that we have such a layer, what shall be build on top of it?   Well, how about  Full-Text Search ?  This is  part 7  of a ongoing series on Graph Databases and Neo4j.   part 1  : Graph Databases (Neo4j) - a revolution in modeling the real world! part 2  : Neo4j Sandbox Tutorial : try Neo4j and learn Cypher the free and easy way part 3  : Neo4j & Cypher Tutorial : Getting Started with a Graph Database and its Query Language part 4  : Using Neo4j with Python : the Open-Source Library NeoAccess part 5  : Using Schema in Graph Databases such as Neo4j part 6 :   Putting it All Together - a Technology Stack on Top of a (Neo4j) Graph Database part 7 : (SPECIAL TOPIC) Full-Text Search with the Neo4j Graph Database part 8 (upcoming!) : (SPECIAL TOPIC) Document Management Full-Text Searching/

PET/CT Combined Scanners - a 2018 Breakthrough of the Year... and a Personal Story

Image source Recently, a co-worker in her 20's was diagnosed with a brain tumor!  At times like these, the importance of medical imaging jumps to the fore! Most people have heard of CT ("CAT") scanners – at least enough to know that they don't actually involve cats – but less well-known are PET scanners (which likewise don't involve pets!), and the synergistic combination of the two. A Marriage Made in Heaven What do those scanners do?  And why are they being combined in single devices? Voted 2018 Breakthrough of the Year by a science magazine , the improved PET/CT combined scanner has been a game changer. The EXPLORER PET/CT scanner – the world’s first medical imaging system that can capture a 3D image of the entire human body simultaneously – has produced its first human images. Developed by UC Davis scientists and a multi-institutional consortium, EXPLORER can scan up to 40 times faster, or use up to 40 times less radiation dose, than

RDF Triple Stores vs. Property Graphs : How to Attach Properties to Relationships

Time for the opening shot of a series about Semantic Technology , and in particular contrasting-and-comparing the opposing (but perhaps ultimately complementary) camps of:   RDF Triple Stores , aka Triples-Based Graphs.   For example, Blazegraph or Apache Jena   (Labeled) Property Graphs .  For example, Neo4j or Blazegraph (For this article, I'll assume that you have at least a passing acquaintance with both.  Here is background info on Triplestores and Property Graphs ) It’s my opinion that modeling in terms of Subject/Predicate/Object triples (aka RDF ) might be appealing to mathematicians or philosophers for its minimalist foundation (though a lot of baroque add-on’s quickly come out of the closet!) Modeling in terms of (Labeled) Property Graphs might be appealing to computer scientists, because such graphs appear more usable and less clunky once you start actually doing something with them. Perhaps because I straddle both the Math and CS camps, I’m currently on t

Brain Microarchitecture : Feedback from Higher-order areas to Lower-order areas

Some questions that arise in Machine Learning involve the prospect of using feedback from Higher-order areas (downstream) to Lower-order areas (upstream), and using Global Knowledge for Local Processing.  A desire to gain insight into those issues from Neuroscience ("how does the brain do it?") led me to some fascinating investigations into the Microcircuits of the Cerebral Cortex.  This blog entry is a broad review of the field, in the context of the original motivating questions from Machine Learning.   Starting out with a quote from the “bible of Neuroscience”: From Principles of Neural Science, 5th edn  (Online book location 1435.3 / 5867).  Emphasis and note added by me: Sensory pathways are not exclusively serial; in each functional pathway higher-order areas project back to the lower-order areas from which they receive input. In this way neurons in higher-order areas, sensitive to the global pattern of sensory input, can modulate the activity of neurons in lowe