Jazz-Packs
Purpose
The purpose of this documentation is to explain the concept of the jazz-packs and to guide developers on how to create and configure these components.
Note: in the near future, the idea is that the jazz-packs will be generated automatically by a graphical user-friendly interface, that will allow users with minimal technical knowledge to be able to create the packs. At this moment, since this interface is still not available, the purpose of this documentation is to allow the developers to move on with creation and configuration of jazz-packs that will cover the most popular demands (like Febraban layout in Brazil for example).
What Is A Jazz-Pack
In order to explain the concept of the jazz-packs, it is important to take a step back and explain the whole ecosystem of Jazz. The Jazz Ecosystem is composed basically by 3 types of components (that in the end are nothing more than node modules): To make it easier to understand, think of it as a car:
jazz-core: this is the main engine. Basically is a task runner that receives the instructions through the jazz-pack and processes the tasks in a pipeline. It also has a single native plugin embedded inside it, that is the Transformer plugin (by the way, this is the reason for the name of the product! If you like Transformers you will understand :-). The Transformer Plugin will be explained moving forward.
jazz-plugins: These are the parts of the car, such as wheels, the multimedia system, the onboard computer, the braking system, and so on. These are the jazz-plugins. The way that Jazz application was designed, it is possible to create plugins or extensions for the most diverse purposes. For example, we can have a plugin that reads data from an excel spreadsheet and saves into a plain javascript object, another plugin that gets a javascript array, and saves the rows into a text file. Each plugin can have one or many tasks. Each task can be configured to receive an unlimited number of input parameters.
jazz-packs: This is the car driver. Or you can think of it as a list of instructions necessary for the processing of the pipeline. Basically, there are 2 main information that need to exist on a jazz-pack file: the pipeline and each plugin configuration.
Technically speaking , the jazz-pack is nothing more than a javascript file, that will export a javascript object, with a bunch of properties.
Get Started
Prerequisites
Note: visit the section Preparing the Environment for more details
- NodeJs v8.11.3 or greather installed
- NPM v5.6.0 or greather installed ( comes with NodeJs )
Clone the project boilerplate:
git clone https://gitlab.com/bruno-bert/jazz-pack-boilerplate jazz-pack-{your_pack_name}
cd into your project directory:
jazz-pack-{your_pack-name}
Install the npm dependencies:
npm install
The naming convention
We highly recommend to use the following naming convention for Jazz-Pack names:
jazz-pack-{packageName}
Example File
Here we have an example of a jazz-pack file. Each of the properties will be explained in a separated section on this documentation.
module.exports = {
pipeline: [
`${excelExtractorPluginPath}:jazz-plugin-excelextractor:extract`,
`${excelExtractorPluginPath}:jazz-plugin-excelextractor:validate`,
`${transformerPluginPath}:Transformer:transform`,
`${textLoaderPluginPath}:jazz-plugin-textloader:load`,
],
plugins: [
{
name: 'jazz-plugin-excelextractor',
sheet: 'source',
},
{
name: 'Transformer',
groupIdColumn: 'groupid',
tasks: {
transform: {
rawDataFrom: 'extract',
},
},
},
{
name: 'jazz-plugin-textloader',
tasks: {
load: {
rawDataFrom: 'transform',
},
},
},
],
userInputParameters: {
customer_country_code: {
alias: 'ccc',
describe: 'Customer Country Code',
demandOption: false,
default: '320',
},
...
},
.pipeline [ ]
This is an array that contains all the steps to be executed. It can have as many steps as you want:
pipeline: [
`${excelExtractorPluginPath}:jazz-plugin-excelextractor:extract`,
`${excelExtractorPluginPath}:jazz-plugin-excelextractor:validate`,
`${transformerPluginPath}:Transformer:transform`,
`${textLoaderPluginPath}:jazz-plugin-textloader:load`
];
In the pipeline above, there are 4 steps:
- extract data from an excel file and save into a javascript object
- validate the extracted data
- transform the data
- load the transformed data into a text file
Important: the steps will executed by Jazz in the sequence they are informed in the array.
Note that the steps will always be tasks. These tasks are ES6 Classes that inherit a Task Class in Jazz core lib. They can be in 3 different places:
- External plugins (that is the case of the steps 1,2 and 4)
- Native plugins (in fact, we just have one that is the Transformer) (that is the case of the step 3)
- Inside the code base (used only during development time of jazz-plugins)
The step name must follow this template:
pluginPath:pluginName:taskId
pluginPath: This is the source of the task code. As mentioned before, it can reference basically to 3 different sources:
- External plugin: let's say you want to test your plugin, by receiving as raw data the result of step executed by a third-party plugin. That is possible by referencing the plugin path from node-modules. Also, the plugin used here needs to be installed in the project as a dev dependency. For example:
```javascript
const defaultRelativePluginPath = "../..";
const textLoaderPluginPath = `${defaultRelativePluginPath}/jazz-plugin-textloader/dist`;
// and in the pipeline step:
`${textLoaderPluginPath}:jazz-plugin-textloader:load`;
```
- Native plugin: In Jazz there is a specific plugin that is native, which means is part of the jazz-core lib. This is the Transformer plugin, which is responsible for receiving a javascript object and convert into another object by following a bunch of conversion rules. This is well explained in section "The Transformer Plugin". It can be used in the pipeline by using the reserved word native. For example:
native: Transformer: transform;
Same codebase: used only in development time. Imagine you are developing a jazz-plugin, you probably will use this option. You can reference a task class in same codebase by using the reserved word this. For example, in case we are developing a plugin that sends emails, we will need to test the task sendmail, you can reference it by using:
this:jazz-plugin-sendmail:sendmail
Important: A single plugin can have as many tasks as it requires to have the work done. It means that each plugin comes with a default internal pipeline. You can configure your jazz-pack to run all tasks inside the plugin by using the following syntax:
pluginPath:pluginName:*
Notice that instead of informing the taskId in the third part, we can add the *. This will make Jazz understand that you want to run all the tasks in the plugin default pipeline.
.plugins [ ]
This is an array of objects. Each object consists in a different plugin. Usually, each plugin that you may have in your pipeline, will need to have an object referenced here. And the reason is that you need to configure the behavior of your plugin, and the tasks inside of the plugin. Let's go the a practical example:
plugins: [
{
name: 'jazz-plugin-excelextractor',
sheet: 'source',
},
{
name: 'Transformer',
groupIdColumn: 'groupid',
tasks: {
transform: {
rawDataFrom: 'extract',
},
},
},
{
name: 'jazz-plugin-textloader',
tasks: {
load: {
rawDataFrom: 'transform',
},
},
},
],
Let's focus on the first object, that represents a plugin:
{
name: 'jazz-plugin-excelextractor',
sheet: 'source',
}
Ok, in here we have 2 properties:
name: the main property of the plugin. This is a required information for ALL plugins, and the name needs to be equal to the name informed in the pipeline.
sheet: the second property is specific to the jazz-plugin-excelextractor. This basically informs the name of the tab in the excel spreadsheet that needs to be read.
Note 1: For this case, there is no property called "tasks". That is ok, it is not required. not having it means that I don't need to setup a specific behavior to specific task in the pipeline.
Note 2: A plugin can have as many properties it requires to have the work done. Q. How do I know what are the available properties for each plugin? A. You need to look into the jazz-plugin documentation.
Note 3: If you are a jazz-plugin developer, it is very important to have a well documented plugin!
Let's move on to the second object in the plugin of arrays:
{
name: 'Transformer',
groupIdColumn: 'groupid',
tasks: {
transform: {
rawDataFrom: 'extract',
},
},
},
This relates to the native plugin Transformer.
Notice that there is the name again (which is always required), and another property now called "groupIdColumn". This is a property specific to the transformer plugin.
Basically, it identifies what it the property name in the source object (the one that will be transformed) that is used for grouping records. This will be well explained in the Transformer Plugin section.
Let's put a magnyfing glass in this particular piece:
tasks: {
transform: {
rawDataFrom: 'extract',
},
}
We have the possibility to modify the behavior of a task in the pipeline, by specifying the tasks through the tasks object, followed by the id of the task (in the case above, we are adding a behavior to the task transform that is inside the plugin Transformer).
Which properties we can setup for the tasks?
These are the properties and methods you can set to ANY tasks (that do not depends on specific plugin behavior), because they relate to the Task Class, that is in the core of the product:
- rawDataFrom: this is the task id of a specific task in the pipeline. Basically it says to the current task from which place it will get the raw data (the data before task processing). In the example above, we are saying to the transform task that it needs to get the raw data from the result of the processing of the task extract.
- getRawData(): this is a method that is usually implemented by the plugin. However, you can ovewrite the behavior of this by setting this on the jazz-pack.
This is the syntax to ovewrite a method:
tasks: {
transform: {
getRawData: () => {
/*set your logic here to return the raw data
once this approach is taken, jazz will NOT look into the rawDataFrom property anymore
example : */
return ['row 1 - this array will be processed by the task',
'row 2 - this array will be processed by the task',
'row 3 - this array will be processed by the task',]
},
},
}
Let's say you do not want to polute your jazz-pack file with a bunch of inline functions. In this case, you can use the syntax below, which will refer to a function from a specific directory of functions.
tasks: {
transform: {
getRawData: require('./functions/getDataToTransform')
},
}
Then, in your project folder, you will have structure below. You just need to make sure your functions directory will be distributed in "dist" folder, and you are good to go:
- execute(): this method is tha main one to the task in the plugin. It is required to be implemented by the plugin. Even this one you can completely change it's default behavior by ovewritting this in the jazz-pack level.
Of course, it provides high level of flexibility, however, one thing you always need to consider is: if you gte yourself changing the plugin behavior too much, maybe you should look for another plugin that would fit better your needs.
- preExecute(): this method is executed before the execute() method. Let's say you want to get the raw data in a step before the execute method, and add something to it. The way to achive this is to ovewritting this method.
Example:
tasks: {
transform: {
preExecute: () => {
/* this is the way to get the data from a specific task in the pipeline
here we are getting the data from extract step */
const defaultRelativePluginPath = '../..';
const pipelinePath = defaultRelativePluginPath + '/jazz-core/dist/Pipeline'
const dataFromExtractStep = require(pipelinePath).getResult("extract");
/* let's say this is my data at this moment:
{
name: 'John',
lastname: 'Rodriguez'
}
Now, let's say that before let the transform task do it's work,
we want to add more information to it, like this:
*/
const newDataFromExtractStep = { ...dataFromExtractStep , email: 'john.rodriguez@jazzmail.com'}
/* now what we have in our new data is:
{
name: 'John',
lastname: 'Rodriguez'
email: 'john.rodriguez@jazzmail.com'
}
Finally, all we have to do is to return the new data as a result from preExecute
*/
return newDataFromExtractStep;
}
/* Important: we want to get the new data in execute
from the preExecute, not from extract.
In this case, we ca change the rawDataFrom as below,
by adding the prefix "pre" and then dash (-) : */
rawDataFrom: 'pre-extract'
},
}
- postExecute(): as you can imagine, this method is executed after the execute() method. The way to work with this is exaclty the same as described on the preExecute(). Except that the prefix used to get the result from postExecute is "post".
rawDataFrom: "post-extract";
- anything else that the plugin developer allows you to ovewrite: Yes! Last but not less important, you can change any other behavior that the plugin developer allows you to change. The plugin developer is able to easily configure what are the properties or methods that can be ovewritten in a jazz-pack level.
.userInputParameters {}
These are parameters used by the Transformer Plugin. Technically speaking, this is a javascript object, that contains other objects. Each of these objects is an input parameter that a final user can inform in the moment they are running a process in Jazz. Typically these basically are behaviors that the user may want to change for different processes they run, by using the same jazz-pack. For example:
userInputParameters: {
customer_account_currency_code: {
alias: 'cacc',
describe: 'Customer Account Currency Code',
demandOption: false,
default: 'GTQ',
},
},
In the example above, the customer_account_currency_code is a field in a source that is being converted by the Transformer Plugin.
In order to understand this, let's go to a real use case:
Imagine that you want to create a jazz-pack that will be responsable for creating a bank file (usually a text file) which will have several fields. In between these fields, you have one that is the country currency, identified as customer_account_currency_code. The bank file format is exactly the same for different countries. You could create a single jazz-pack for each country. Sure, that is one approach to follow, however, there is a better option.
By adding the user input parameter above, you provide to your jazz-pack the flexibility to create an output file with different content, depending on the parameter.
In this case, if you are using the jazz command line tool, and type:
jazz --cacc GTQ
You can have an output file like this, that will fit the need to Guatemala.
GTQ00000000000000000001000000
And by typing:
jazz --cacc BRL
You can have an output file like this, that will fit the need to Brazil.
BRL00000000000000000001000000
Just another example: Think for example in a plugin that send mails. In the user input parameters we could have the smtp host, mail credentials, port, etc...
These are the properties of each user input:
- alias: this is the optio user will type in cli, as explained before
- describe: it's the meaning of the parameter (will be perfect in a graphical user interface, in the future)
- demandOption (true/false): indicates if parameter is required or not
- default: the default value if user do not inform the value
Understand the Boilerplate Project
In this section, we will explain in details every single file on the boilerplate project.
package.json
dependencies
There is no single dependence for the jazz-pack. Remember that this is nothing more than a file that exports a javascript object. And this file will be referenced by the main jazz application.
Because of this, there are only dev Dependencies on this boilerplate, and they are required simply because during development of the jazz-pack, you will need to test your package. So you need a way to run the pipeline.
devDependencies
Contains the dependencies of development of the plugin. These dependencies are used only during development. They are not distributed with the pack.
"devDependencies": {
"@babel/runtime": "^7.4.5",
"@babel/cli": "^7.4.4",
"@babel/core": "^7.4.5",
"@babel/node": "^7.4.5",
"@babel/plugin-transform-runtime": "^7.4.4",
"@babel/preset-env": "^7.4.5",
"babel-plugin-root-import": "^6.2.0",
"del": "^4.1.1",
"eslint": "^5.16.0",
"eslint-config-airbnb-base": "^13.1.0",
"eslint-import-resolver-babel-plugin-root-import": "^1.1.1",
"eslint-plugin-import": "^2.17.3",
"eslint-plugin-jest": "^22.7.1",
"eslint-plugin-promise": "^4.1.1",
"gulp": "^4.0.2",
"gulp-jsonminify": "^1.1.0",
"gulp-uglify": "^3.0.2",
"jazz-core": "^1.0.5",
"jazz-plugin-excelextractor": "^1.0.3",
"jazz-plugin-textloader": "^1.0.5",
"jest": "^24.8.0",
"prettier-eslint-cli": "^4.7.1"
}
This project contains basically 4 groups of dependencies:
Babel core and plugins: used in the process of build, to transpile es6+ code into "legacy javascript code"
Gulp core and plugins: task runner used in the process of build and distribution of the package
Eslint: used to format code by using best practices. In this case, it is using the air-bnb guides.
cryptr: this is the node module used to encrypt / decrypt the jazz-pack file.
Jazz Dependencies:
Important: Always keep in mind that these dependencies must NOT be distributed with your final package. They are here only for your development time, so that you can test your package running in a real pipeline.
jazz-core: it is the main engine of jazz. During development it is referenced in node-modules directory. However, once the plugin is distributed, it will reference the jazz-core lib from the client. This means, it will not be distributed with the pack.
jazz-plugin-textloader: in this boilerplate, it is used only for testing the package end-to-end, by receiving data from another step in the pipeline, that is executed by another plugin. Also, used only for testing purposes during development flow. For this reason, it is referenced in the devDependencies, not in main dependencies. This means, it will not be distributed with the jazz-pack.
jazz-plugin-excelextractor: in this boilerplate, it is used only for testing the package end-to-end, by reading an excel file and sending resulted data to another step in the pipeline, that is executed by another plugin. Also, used only for testing purposes during development flow. For this reason, it is referenced in the devDependencies, not in main dependencies. This means, it will not be distributed with the jazz-pack.
npm scripts
"scripts": {
"build": "babel src -d build && gulp minifyJson",
"prebuild": "gulp cleanBuild",
"lint": "eslint 'src/**/*.js' --fix",
"git": "git add . && git commit -m %RANDOM% && git push",
"test": "jest --watch",
"dist": "npm run build && gulp dist",
"encrypt": "gulp encrypt",
"start": "node ./build/__tests__/run-cli.js --configFile ./build/pack-config --sourceFile ./build/__tests__/source.xlsx --outputFile ./build/__tests__/target.txt",
"dev": "nodemon --exec babel-node ./src/__tests__/run-cli.js --configFile ./src/pack-config --sourceFile ./src/__tests__/source.xlsx --outputFile ./src/__tests__/target.txt --debug",
"dev-enc": "nodemon --exec babel-node ./src/__tests__/run-cli.js --configFile ./src/pack-config.sec --sourceFile ./src/__tests__/source.xlsx --outputFile ./src/__tests__/target.txt --debug --decryptKey JazzIsAwesome"
},
- build: used to transpile the code into legacy javascript code using babel 7. Saves the code into build folder
- prebuild: called automatically by npm before each build. in this case it excludes all the files inside the build directory
- lint: used to run the eslint checks
- git: used to commit and push changes into a git repository
- start: runs the run-cli.js file in "production mode"
- test: can be used for unit testing, if you use decide to use a tool like jest (which we highly recommend)
- dev: runs the run-cli.js file by using nodemon. This improves the development flow, since on each change, it will re-run automatically. This command calls the command line
jazz
with some input parameters:- configFile: references to the jazzpack file (in this case ./src/pack-config)
- outputFile: input parameter specific for the plugin jazz-plugin-textloader. Represents the file path that text file will be saved after the task is precessed
- dist: calls the build and then the task gulp dist. The task dist will perform the following tasks in the sequence: (see gulpfile.js file):
- cleanDist: excludes all files from dist folder
- minifyJson: minifies the json files that may exist in the .src folder
- uglifyJs: uglifies the js files that are in the build folder and saves into dist folder
- replaceDist: some files may have a different version on development time than on productin. This task replaces some files that need to be different on distribution folder.
- encrypt: if you decide to encrypt your pack, you simple need to run this command after the dist command
The __tests__ directory
All files inside this folder will be excluded from distribution package. They are used only during development time, for testing purposes.
run-cli.js
This file contains only 2 lines of code. It calls the run method inside the CliLoader class. The CliLoader class is available in the jazz-core lib, and is responsible for creating a jazz command line tool.
The run method is responsible for starting the pipeline. It means it will read the jazz-pack file, mount the pipeline with all the tasks and call each task in the sequence.
import CliLoader from "jazz-core/dist/CliLoader";
CliLoader.run();
By running the bash npm run dev
, it will make possible testing the jazz-pack inside the pipeline.
Note: for running the pipeline, you need to have the jazz-core and the jazz-plugins used installed in your project as a devDependency. That is the case of this boilerplate project:
"devDependencies": {
"jazz-core": "^1.0.9",
"jazz-plugin-excelextractor": "^1.0.3",
"jazz-plugin-textloader": "^1.0.5",
}
Distribute your Jazz-Pack
First thing to do is to pack your jazz-pack. :-( This boilerplate already has a npm script that will do the work for you. All you need to do is to open the terminal and on the project folder, type:
npm run dist
On package.json file you can see that this command will run the commands below, in sequence:
npm run build && gulp dist
npm run build: the build command will run the babel transpiler (using version 7) that will convert your code from new javascript (+ ES6) to "old" javascript (< ES6).
gulp dist: when opening the gulfile.js, the dist task does the following:
- cleanDist: excludes all files from dist folder
- minifyJson: minifies the json files that may exist in the .src folder
- uglifyJs: uglifies the js files that are in the build folder and saves into dist folder
- replaceDist: some files may have a different version on development time than on production. This task replaces some files that need to be different on distribution folder.
Ok, so now you tested your package, created the distribution package and you are good to go for production. In soon future, the idea is to have an specific marketplace for jazz-packs and jazz-plugins in which the users will be able to download free or buy the plugins and packs (almost like the VSCode market place of extensions). At this moment, this feature is not developed, so that, the only way to publish the jazz packs is through well known npm.
So, as a pre-requisite, you need to have an account created on https://www.npmjs.com.
Then, all you need to do is to run the command in the terminal:
npm publish
The .npmignore file will guarantee that only the dist folder will be published. Of course, if your intention is to charge for the pack, by publishing it on npm is still not the best solution. As mentioned, the Jazz MarketPlace is a work in progress.
Perfect, your jazz-pack is developed and available for others to use.
Encrypt Your Jazz-Pack
If you want to encrypt your jazz-pack (which will hide your code form end users), you can type the command:
npm run encrypt
This is the same to run the gulp encrypt command, which will run the gulp task named encrypt (check the encrypt function inside the gulpfile.js)
When you look at the gulpfile.js, you can see that this task gets the pack-config.js file (the original) and creates a pack-config.sec file (the encrypted). And by the end, deletes the original file from the dist folder (of course, you won't want the original file there, only the encrypted one).
const configFile = "./src/pack-config.js";
const encryptedFile = "./src/pack-config.sec";
Of source, if you decide to change the name of the pack-config.js file, that is ok, but keep in mind that you will need to change the npm scripts and the gulpfile.js, that have references to the pack-config.js file.
Do I need to encrypt the pack?
Well, as explained, the boilerplate project will only provide you an easy way to perform the encryption. However, it's up to you if you will protect your package code or will keep this open source.
There are 2 options:
You can release the package and provide users the source code, which will allow anyone to change your original configuration and use for their own purpose. In this case, encryption will not be necessary;
depending on complexity of your package, you could develop it and charge for the downloads in future jazz platform (to be developed). In this case, that will be a good idea to encrypt your jazz-pack.
Note: at this moment, the feature of encryption that will keep your code safe is not that "safe" yet. This capability still needs to be improved. In the future, once we have a market place to distribute the packages, the users will need to be logged in to download and install the packages. In this moment, we will be able to improve the feature.