Here is my presentation from the SmartData / NoSQL Now! conference today:
MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Here is my presentation from the SmartData / NoSQL Now! conference today:
MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
I’m excited for my talk tomorrow at the NoSQLNow! conference! I hope those in the San Francisco/San Jose area can make it, and for those that do, please join me on Tuesday at 2pm!
— Marc Hadfield
Further details:
Tuesday, August 18, 2015
02:00 PM – 02:45 PM
SmartData, NoSQL Now! conference in San Jose, California
SmartData conference: http://smartdata2015.dataversity.net/
NoSQL Now conference: http://nosql2015.dataversity.net/
MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Each database has its strengths and weaknesses for different data access profiles, and we should endeavor to use the right tool for the right job.
However, adding another infrastructure component greatly increases not only the management effort, but also the development effort to integrate and maintain connections across multiple data repositories, let alone keeping the data synchronized.
In this talk, we’ll discuss MetaQL, a common query layer across database technologies including NoSQL, SQL, Sparql, and Spark.
Using a common query layer lessens the burden on developers, allows using the right database for the right job, and opens up data to additional analysis that would be unavailable previously – providing new and unexpected value.
In this talk, we will discuss:
More details: http://nosql2015.dataversity.net/sessionPop.cfm?confid=90&proposalid=7823
Along with our recent release of VDK 0.2.254, we’ve added a few new example apps to help developers get started with the VDK.
By starting with one of these examples, you can quickly build applications for prediction, classification, and recommendation with a JavaScript web application front end, and prediction models on the server. The examples use prediction models trained using Apache Spark or an external service such as AlchemyAPI (IBM Bluemix), or Metamind.io.
There is also an example app for various queries of a document database containing the Enron Email dataset. Some details on this dataset are here: https://www.cs.cmu.edu/~./enron/
The example applications have the same architecture.
The components are:
Here is a quick overview of some of the examples.
We’ll post detailed instructions on each app in followup blog entries.
MetaMind Image Classification App:
Source Code:
https://github.com/vital-ai/vital-examples/tree/master/metamind-app
Demo Link:
https://demos.vital.ai/metamind-app/index.html
Screenshot:
This example uses a MetaMind ( https://www.metamind.io/ ) prediction model to classify an image.
AlchemyAPI/IBM Bluemix Document Classification App
Source Code:
https://github.com/vital-ai/vital-examples/tree/master/alchemyapi-app
Demo Link:
https://demos.vital.ai/alchemyapi-app/index.html
Screenshot:
This example app uses an AlchemyAPI (IBM Bluemix) prediction model to classify a document.
Movie Recommendation App
Source Code (Web Application):
https://github.com/vital-ai/vital-examples/tree/master/movie-recommendations-js-app
Source Code (Training Prediction Model):
https://github.com/vital-ai/vital-examples/tree/master/movie-recommendations
Demo Link:
https://demos.vital.ai/movie-recommendations-js-app/index.html
Screenshot:
This example uses a prediction model trained on the MovieLens data to recommend movies based on a user’s current movie ratings. The prediction model uses the Collaborative Filtering algorithm trained using an Apache Spark job. Each user has a user-id such as “1010” in the screenshot above.
Spark’s collaborative filtering implementation is described here:
http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html
The MovieLens data can be found here:
http://grouplens.org/datasets/movielens/
Enron Document Search App
Source Code:
https://github.com/vital-ai/vital-examples/tree/master/enron-js-app
Demo Link:
https://demos.vital.ai/enron-js-app/index.html
Screenshot:
This example demonstrates how to implement different queries against a database, such as a “select” query — find all documents with certain keywords, and a “graph” query — find documents that are linked to users.
Example Data Visualizations:
The Cytoscape graph visualization tool can be used to visualize the above sample data using the Vital AI Cytoscape plugin.
The Cytoscape plugin is available from:
https://github.com/vital-ai/vital-cytoscape
An example of visualizing the MovieLens data:
An example of visualizing the Wordnet Dataset, viewing the graph centered on “Red Wine”:
For generating and importing the Wordnet data, see sample code here:
Information about Wordnet is available here:
https://wordnet.princeton.edu/
Another example of the Wordnet data, with some additional visual styles added:
VDK 0.2.254 was recently released, as well as corresponding releases for each product.
The new release is available via the Dashboard:
Artifacts are in the maven repository:
https://github.com/vital-ai/vital-public-mvn-repo/tree/releases/vital-ai
Code is in the public github repos for public projects:
https://github.com/orgs/vital-ai
Highlights of the release include:
Vital AI Development Kit:
VitalPrime:
Aspen:
Vital Utilities:
Vital Vertx and VitalService-JS:
Vital Triplestore: