Skip to content
eucalypt
v0.15.0

Working with Data

In this chapter you will learn:

  • How to process JSON, YAML, TOML, CSV, and XML data
  • How to convert between formats on the command line
  • Patterns for querying and transforming structured data
  • How to combine multiple data sources

The simplest use of eucalypt is converting between data formats. By default, output is YAML:

Terminal window
# JSON to YAML
eu data.json
# YAML to JSON
eu data.yaml -j
# JSON to TOML
eu data.json -x toml

Pipe JSON from other tools into eucalypt:

Terminal window
curl -s https://api.example.com/users | eu -e 'map(.name)'

Or process a JSON file:

Terminal window
eu -e 'users filter(.active) map(.email)' data.json

YAML files are read natively. All YAML features including anchors, aliases, and merge keys are supported:

config.yaml
defaults: &defaults
timeout: 30
retries: 3
production:
<<: *defaults
debug: false
Terminal window
eu config.yaml -e 'production'
timeout: 30
retries: 3
debug: false

YAML timestamps are automatically converted to date-time values:

created: 2024-03-15
updated: 2024-03-15T14:30:00Z

Quote the value to keep it as a string: created: "2024-03-15".

Terminal window
eu config.toml -e 'database.port'
5432

CSV files are imported as a list of blocks, where each row becomes a block with column headers as keys:

Terminal window
eu -e 'rows filter(_.age num > 30)' rows=people.csv

CSV values are always strings. Use num to convert to numbers when needed.

XML is imported as a nested list structure. Each element is represented as [tag, attributes, ...children]:

Terminal window
eu -e 'root' root=xml@data.xml

Use list functions to navigate the structure:

{ import: "root=xml@data.xml" }
# Get the tag name (first element)
tag: root first
# Get attributes (second element)
attrs: root second
# Get child elements (everything after the first two)
children: root drop(2)

Use named inputs to make data available under a specific name:

Terminal window
eu users=users.json roles=roles.json -e 'users map(.name)'

Named inputs are essential for list-based formats (CSV, JSON Lines, text):

Terminal window
eu lines=text@log.txt -e 'lines filter(str.matches?("ERROR")) count'

A powerful pattern is combining data from multiple sources:

Terminal window
eu users.yaml roles.yaml merge.eu

Where merge.eu contains logic that uses names from both inputs:

merge.eu
summary: {
user-count: users count
role-count: roles count
}

The -e flag specifies an expression to evaluate against the loaded inputs:

Terminal window
# Select a nested value
eu config.yaml -e 'database.host'
# Transform and filter
eu data.json -e 'items filter(.price > 100) map(.name)'
# Aggregate
eu data.json -e 'items map(.price) foldl(+, 0)'

The --collect-as (-c) flag gathers multiple files into a list:

Terminal window
eu -c configs *.yaml -e 'configs map(.name)'

Add --name-inputs (-N) to get a block keyed by filename:

Terminal window
eu -c configs -N *.yaml
configs:
a.yaml:
name: alpha
b.yaml:
name: beta

Control the output format:

Flag Format
(default) YAML
-j JSON
-x json JSON
-x toml TOML
-x edn EDN
-x text Plain text

The format can also be inferred from the output file:

Terminal window
eu data.yaml -o output.json

Suppose you have a CSV of sales data and want to generate a JSON summary:

Terminal window
eu sales=sales.csv -j -e '{
total: sales map(.amount num) foldl(+, 0)
count: sales count
regions: sales map(.region) set.from-list set.to-list
}'

Or as a reusable eucalypt file:

report.eu
{ import: "sales=sales.csv" }
` :suppress
amounts: sales map(.amount num)
report: {
total: amounts foldl(+, 0)
count: sales count
average: report.total / report.count
}
Terminal window
eu report.eu -j -e report
  • Eucalypt reads JSON, YAML, TOML, CSV, XML, EDN, JSON Lines, and plain text
  • Output defaults to YAML; use -j for JSON or -x for other formats
  • Named inputs (name=file) give data a name for reference
  • The -e flag evaluates expressions against loaded data
  • --collect-as gathers multiple files into a list or block
  • CSV values are strings; use num to convert to numbers
  • Combine multiple sources with the command line input system or imports