Creating a predictive model training set with Vital AI

Part of an ongoing series to introduce the Vital AI software used to make predictions.

Go to beginning: Using the Vital AI software to make predictions

To process data with a machine learning algorithm to build a predictive model, a dataset must be created.

The Twenty Newsgroup source data is comprised of around 20,000 individual text files – one per article.

The Vital AI software uses a standardized data format for datasets, with each data object conforming to the data model.

To convert the source data into the Vital AI data format, we use a simple script.

The key lines of the script are:

...

def doc = new TwentyNewsDocument()

doc.URI = "http://example.org/twentynews/${newsgroup}/${id}";

doc.title = subject

doc.body = body

doc.newsGroup = 'http://vital.ai/twentynews/Category/' + newsgroup;

writer.startBlock();

writer.writeGraphObject(doc);

writer.endBlock();

}

The resulting data file is in the “Vital Block” format, called “block” format as data objects can be grouped together in “blocks” for processing.

Next: Processing datasets using Machine Learning on Hadoop with Vital AI