JSON Tools
Cleanse and filter JSON for later use.
{{ jsonTools | filter: "value" | filterType: "" }}
Parameters:
- Filter: A value for which we should filter items.
- Filter Type: If set to
Equals, filter for objects/keys where the value equals the value set infilter. If set toNot Equals, filter for objects/keys where the value does not equal the value set infilter.
Returns: The resulting JSON.
Example:
{
"books": [
{
"title": "The Great Gatsby",
"summary": "The Great Gatsby is a novel by F. Scott Fitzgerald that follows the story of Jay Gatsby, a wealthy and mysterious man, and his pursuit of the American Dream. Set in the 1920s, the book explores themes of love, wealth, and the corruption of the American Dream.",
"author": "F. Scott Fitzgerald"
}
]
}
{{ jsonTools | filter: "The Great Gatsby" | filterType: "Equals" }}
Would yield:
{
"books": [
{
"title": "The Great Gatsby"
}
]
}
Where setting filterType to Not Equals would yield:
{
"books": [
{
"summary": "The Great Gatsby is a novel by F. Scott Fitzgerald that follows the story of Jay Gatsby, a wealthy and mysterious man, and his pursuit of the American Dream. Set in the 1920s, the book explores themes of love, wealth, and the corruption of the American Dream.",
"author": "F. Scott Fitzgerald"
}
]
}
JSON to Variables
Accepts JSON as an input, flattens the object keys, and sets those keys as variables in mAIstro's context.
Parameters: None - Data should be "chained" into this function.
Returns: None - Variables are assigned as a result of this function.
Example: Data can come from a LLM, a file, REST API response, etc
{{ LLM | prompt: "Output some information about a book in JSON format. Include title, summary, and author." | modelCard: "" }}=>{{ jsonToVars }}
The output from the LLM:
{
"title": "The Great Gatsby",
"summary": "The Great Gatsby is a novel by F. Scott Fitzgerald that follows the story of Jay Gatsby, a wealthy and mysterious man, and his pursuit of the American Dream. Set in the 1920s, the book explores themes of love, wealth, and the corruption of the American Dream.",
"author": "F. Scott Fitzgerald"
}
And finally, looking in the variable inspector, you can see the variables now set available for use:
Variables to JSON
Convert environment variables into JSON.
{{ varsToJSON | path: "" | variable: "" }}
Parameters:
- Path: (Optional) The flattened path from which to start obtaining values.
- Variable: The name of the variable to assign the resulting JSON.
Returns: Nothing - Output is assigned to the set variable name.
Example 1:
Howard has 20 cats and 40 dogs.
He took them to the vet last week.=>{{ grammar }}=>{{ variable | name: "text" }}
{{ varsToJSON | path: "" | variable: "gm" }}
<< name: gm, prompt: false >>
Will yield:
{
"grammar": {
"year": [
2024
],
"context": "",
"dates": [
"last",
"week"
],
"propernouns": [
"Howard"
],
"nouns": [
"20 cats",
"40 dogs",
"vet",
"week"
],
"preps": [
"He",
"them"
],
"determiners": []
},
"text": "Howard has 20 cats and 40 dogs. \nHe took them to the vet last week."
}
Example 2:
Using the path parameter, we can specify the starting path of values we want:
Howard has 20 cats and 40 dogs.
He took them to the vet last week.=>{{ grammar }}=>{{ variable | name: "text" }}
{{ varsToJSON | path: "grammar.dates" | variable: "gm" }}
<< name: gm, prompt: false >>
Will yield:
{
"grammar": {
"dates": [
"last",
"week"
]
}
}
Summarize
Summarizes input text while preserving the main subject of the content.
{{ summarize|length:100|match:"" }}
Parameters:
- Length: The total maximum character length of the output/summary.
- Match: The text around which to prioritize the summary.
Returns: The resulting summary.
Example 1:
I have 20 cats and 40 dogs - it's a lot of furry friends to take care of!
My name is Jane and I run an animal rescue shelter out of my home.
It all started a few years ago when I took in a litter of abandoned kittens.
I fell in love with them and decided to make it my mission to give unwanted animals a forever home.
{{ summarize|length:100 }}
Yields:
I have 20 cats and 40 dogs - it's a lot of furry friends to take care of!
Example 2: Using match
I have 20 cats and 40 dogs - it's a lot of furry friends to take care of!
My name is Jane and I run an animal rescue shelter out of my home.
It all started a few years ago when I took in a litter of abandoned kittens.
I fell in love with them and decided to make it my mission to give unwanted animals a forever home.
{{ summarize|length:100|match:"love" }}
Yields:
I fell in love with them and decided to make it my mission to give unwanted animals a forever home.
Truncate by Tokens
Helpful to manage the size of context sent to the LLM, this allows you to truncate to a specific number of tokens effortlessly.
{{ truncateToken | tokens: "" }}
Parameters:
- Tokens: The maximum number of tokens to allow in the returned text.
Returns: The resulting text clipped to the specified amount of tokens.
Example 1:
{{ kb | query: "NeuralSeek" }}=>{{ truncateToken | tokens: "2000" }}=>{{ variable | name: "documentation" }}
Would yield a variable far too large to include here, but would limit the resulting documentation text to 2000 (2k) tokens before assigning to the documentation variable. This helps prevent exceeding context windows of some smaller LLMs.
Remove Stopwords
Removes stop words from input text.
{{ stopwords }}
Parameters: None - Data should be "chained" into this function.
Returns: The resulting text with stopwords removed.
Example:
I have 20 cats and 40 dogs, isn't this amazing?
{{ stopwords }}
Will yield
20 cats 40 dogs, amazing?
Notice the words I, have, and, isn't, this are deemed as stopwords and thus have been removed.
Force Numeric
This function removes all non-numeric characters, and string-style concatenates the remainder into a single value.
{{ forceNumeric }}
Parameters: None - Data should be "chained" into this function.
Returns: The resulting number.
Example:
I have 20 cats and 40 dogs contains numeric values. So, running this:
I have 20 cats and 40 dogs
{{ forceNumeric }}
Will yield: 2040
Table Prep
This function prepares tabular data to be better understood and processed by LLM.
{{ tablePrep | query:"" | sentences: "true" }}
Parameters:
- Query: Keywords to help narrow the returned data.
- Sentences: If true, return the output in natural language expressions. If false, return JSON format.
Returns: The resulting natural language text or JSON.
Example 1:
If we have CSV data, table prep will convert it to JSON or natural language:
col1,col2,col3
data1,data2,data3
data11,data22,data33
{{ tablePrep | sentences: "false" }}
The result will be:
{
"col1": [
"data1",
"data11"
],
"col2": [
"data2",
"data22"
],
"col3": [
"data3",
"data33"
]
}
Example 2: Using the query parameter:
col1,col2,col3
data1,data2,data3
data11,data22,data33
{{ tablePrep|query: "values for col1" }}
Will yield all the values for col1:
{
"col1": [
"data1",
"data11"
]
}
Example 3: Using the sentences: true parameter:
col1,col2,col3
data1,data2,data3
data11,data22,data33
{{ tablePrep | sentences: "true" }}
Will yield:
Record number 0 lists that col1 is data1, and the col2 is data2, and the col3 is data3.
Record number 1 lists that col1 is data11, and the col2 is data22, and the col3 is data33.
Split
Simple split operation to split (or cut) part of text using start and end string.
Alternatively used to remove page headers/footers. Helpful when processing web content/text.
{{ split | start: "" | end: "" | removeHeaders: true }}
Parameters:
- Start: Match string to begin the split. Included in the result. Case-sensitive.
- End: The match string to end the split. Excluded from the result. Case-sensitive.
- Remove Headers: If true, remove repeating lines of text (e.g. headers or footers). If false, do not remove repeating text.
Returns: The resulting split chunk of text.
Example 1:
I have 20 cats and 40 dogs.
{{ split | start: "20" | end: "40" | removeHeaders: false }}
will yield:
20 cats and
Example 2:
Using removeHeaders will strip frequently repeating lines out of given input text:
My animals:
I have 20 cats and 40 dogs.
My animals:
I have 47 hamsters and 22 pet snakes.
My animals:
I pet all my animals every day.
{{ split | removeHeaders: true }}
Would yield:
I have 20 cats and 40 dogs.
I have 47 hamsters and 22 pet snakes.
I pet all my animals every day.
Regular Expression
Performs regular expression on input data. Regular expression can be a powerful feature to extract or replace certain data.
{{ regex | match: "" | replace: "" | group: "" }}
Parameters:
- Match: The match regex to use. E.g.
/[^0-9A-Za-z\s]/g - Replace: The string to substitute for matches.
- Group:
Returns: The replaced text, or in case of using the group parameter, the group match.
Example 1:
If you have a text that you need to replace with something else, you can use the following expression:
my name is howardyoo
{{ regex | match: "yoo" | replace: "yu" }}
Which yields:
my name is howardyu
Example 2:
Regex also supports extraction. For example, if you want to extract the email address in a text message, you can do so:
my name is [email protected]
{{ regex | match: "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}" | group: "0" }}
This will extract the email address (group 0). The result is:
Example 3:
Regex also supports groups, so in case you want to get the last digits of a phone number, you can do so:
my phone number is 213-292-3322
{{ regex | match: "([0-9]+)-([0-9]+)-([0-9]+)" | group: "3" }}
which will result in:
3322