Embedded Domain Specific Languages in F# using Custom Operations

10 Oct 2014

Introduction

I've written before about F# and embedded domain specific languages. This time we're going to explore a less widely known feature of F#, custom operations within computation expressions, and see how they can help us to build up a simple eDSL for describing NuGet packaging. All the code for this blog post can be found on GitHub.

Custom Operations

Custom operations, not to be confused with custom operators, were added to F# 3.0 and allow for custom keywords and syntax to be defined within a computation expression. Custom operations are the mechanism through which query expressions are implemented.

let results = 
    query {
        for customer in db.Customers do
        where (customer.Name = "smith")
        select customer
    }

A simple NuGet eDSL

We'll break the problem down into three stages. First, we'll define a small set of data structures that represents all the information we need in order to build a NuGet package. Second, we'll define a NuGet computation expression, along with custom operations that represents our eDSL, and use this to construct our data structures. Thirdly, we'll pass this data structure to a separate function that will actually perform the NuGet packaging operation according to the specification defined by our eDSL.

Let's start by defining our data types. We'll need two main types, one to hold information about the package itself (e.g. package ID, version) and another to hold data about the packaging process (e.g. where to find the NuGet executable, where to output the package). We'll also define some ancillary and convenience types too.

// We're referencing the NuGet core package in order to get access to their
// version data structure and parsing functionality.
type version = NuGet.SemanticVersion

// We'll just store the package ID and version for now. We'll expand this
// shortly
type PackageDefinition = 
    {
        id : string
        version : version
    }

// We'll also need to store information about where to find the nuget.exe, the
// solution's root folder and an output folder. Most importantly, we'll need a
// list of package definitions from which we'll generate the packages.
type type NuGetDefinition = 
        {
            rootDirectory : string
            toolsDirectory : string
            outputDirectory : string
            projects : PackageDefinition list
        }

It might also be useful to define functions that provide a minimal set of defaults for these data structures:

let defaultPackageDefinition = 
    {
        id = ""
        version = version.Parse("0.0.1")
    }

let defaultNuGetDefinition = 
    {
        projects = []
        rootDirectory = "."
        toolsDirectory = ".//.nuget//"
        outputDirectory = ".//.output//"
    }

Given this extremely minimal definition of the data we need in order to create a package we can now create an equally minimal DSL that allows us to build up these data structures. The DSL will take the form of two computation expression builders, each of which defines custom operations that represent the language of our DSL.

type PackageDefinitionBuilder() = 

    member __.Yield (item:'a) : PackageDefinition = defaultPackageDefinition

    [<CustomOperation("id")>] 
    member __.Id (spec, x) = {spec with id = x }

    [<CustomOperation("version")>] 
    member __.Version (spec : PackageDefinition, x : version) = 
        {spec with version = x}


let package = PackageDefinitionBuilder()

type NuGetDefinitionBuilder() = 
    member __.Yield (item:'a) : NuGetDefinition = defaultNuGetDefinition

    [<CustomOperation("packageProject")>] 
    member __.packageProject (spec, x) = 
        { spec with projects = x :: spec.projects}

    [<CustomOperation("rootDirectory")>] 
    member __.RootDirectory (spec,x) = {spec with rootDirectory = x}

    [<CustomOperation("toolsDirectory")>] 
    member __.ToolsDirectory (spec,x) = {spec with toolsDirectory = x}

    [<CustomOperation("outputDirectory")>] 
    member __.OutputDirectory (spec,x) = {spec with outputDirectory = x}

let nuget = NuGetDefinitionBuilder()

The F# language specification requires that in order to define custom operations the computation expression builder must also define the Yield operation. The Yield function is invoked immediately upon entering the computation expression so we're using it to setup the defaults. These default values then get threaded through the remaining computation expression so that we can make modifications to build up the package definition. This is how we'd use our DSL so far.

let nugetDef = 
    nuget {
        rootDirectory "c:\\dev"
        toolsDirectory "c:\\dev\\tools"
        outputDirectory "c:\\dev\\output"
        packageProject {
            id "Foo.Bar"
            version (v"1.2.3")
        }
    }

At the moment this doesn't actually do anything useful, we'll need to add more operations in order to collect all the data we need to build a useful NuGet package.

One of the key features of NuGet is the ability to specify dependencies between packages. Here's the definition of a DepdendencyDefinition structure that we can use to define package dependencies.

//Just an example subset of versions
type FrameworkVersion = 
    | Net35
    | Net40
    | Net40ClientProfile
    | Net40Compact
    | Net40Full
    | Net45
    | Silverlight3
    | Custom of string

type DepedencyDefinition =
    {
        name : string;
        version :version;
        frameworkVersion : FrameworkVersion option;
    }

We can now extend our DSL to allow for package dependencies to be expressed.

type PackageDefinition = 
    {
        id : string
        version : version
        includedDependencies : DepedencyDefinition list
    }

type PackageDefinitionBuilder() = 

    member __.Yield (item:'a) : PackageDefinition = defaultPackageDefinition

    [<CustomOperation("id")>] 
    member __.Id (spec, x) = {spec with id = x }

    [<CustomOperation("version")>] 
    member __.Version (spec : PackageDefinition, x : version) = 
        {spec with version = x}

    [<CustomOperation("includeDependency")>] 
    member __.IncludeDependency (spec, x) = 
        {
            spec with includedDependencies = x :: spec.includedDependencies
        }

And here's how it could be used.

let nugetDef = 
    nuget {
        rootDirectory "c:\\dev"
        toolsDirectory "c:\\dev\\tools"
        outputDirectory "c:\\dev\\output"
        packageProject {
            id "Foo.Bar"
            version (v"1.2.3")
            includeDependency (
                { name = "xunit";
                  version = v"1.9.1";
                  frameworkVersion = (Some Net40)
                })
        }
    }

We can go on adding more and more package dependencies in a similar way. Constructing dependencies in this way, using record syntax, is not the most friendly though. With a few helper functions we can clean it up to make it much nicer to use, and also allow for optional arguments to be supplied or omitted without having to use the full record syntax (see here for an explanation of how static type constraints work).

// This is a declaration of an infix operator that will invoke an implicit
// conversion from one type to another, as long as the impicit operator is
// defined for either type.
let inline (!>) (x:^a) : ^b = 
    ((^a or ^b) : (static member op_Implicit : ^a -> ^b) x)

type DepedencyDefinition =
    {
        name : string;
        version :version;
        frameworkVersion : FrameworkVersion option;
    }

    static member private create  n v f = 
        {
            name = n; 
            version = v;
            frameworkVersion = f
        }

    static member op_Implicit( x : string * version) = 
        match x with | (n, v) -> DepedencyDefinition.create n v None

    static member op_Implicit( x : string * version * FrameworkVersion) = 
        match x with | (n, v, f) -> DepedencyDefinition.create n v (Some f)

We defined a new infix operator that will enable us to easily cast one type into another using the !> syntax. We then declared a set of implicit conversion for the DependencyDefinition type which accept a tuple of different lengths and return us a DependencyDefinition structure. We can use this new syntax and rewrite our original usage of dependency definitions like this

let nugetDef = 
    nuget {
        rootDirectory "c:\\dev"
        toolsDirectory "c:\\dev\\tools"
        outputDirectory "c:\\dev\\output"
        packageProject {
            id "Foo.Bar"
            version (v"1.2.3")
            includeDependency !> ("xunit", v"1.9.1", Net40)
            includeDependency !> ("autofac", v"1.0.0")
        }
    }

We can continue adding more and more features to our DSL that support all of the packaging options that we want or need. Ultimately ending up with something that allows a rich set of packaging options to be defined.

let nugetDef = nuget {            
       rootDirectory "c:\\dev"
       toolsDirectory "c:\\dev\\tools"
       outputDirectory "c:\\dev\\output"
       packageProject (package {
           projectName "package.test1"
           includeReferencedProjects
           id "package.test"
           version (v"1.2.3")
           authors ["me"; "you"]
           description "package description"
           summary "package summary"
           copyright "package copyright"
           includeFiles !> "bin\**\*.config"
           includeFiles !> ("tool\**\*.exe", "tools", "tool\**\*.pdb")
           includeDependency !> ("xunit", v"1.9.1")
           includeDependency !> ("fsunit.xunit", v"1.3.0.1")
           includeDependency !> ("Autofac", v"1.0.0.0", Net40)
           prerelease
           requireLicenceAccept
       })
       packageProject (package { 
           id "package.test2" 
           // More packaging options omitted for brevity
       })

Once the packaging definition has been built up we can then pass it to another function, that we'd have to write, to actually build the nuspec xml and call the nuget.exe in order to do the packaging. A minimal, but complete, solution which implements these ideas and also the actual task of building the packages can be found on here.

Further Reading

You can get the full source from this article on GitHub. The rules that govern exactly how custom operations get expanded are quite complex, there's many additional options that we can use to tweak the way we express our eDSL which I'll explore in future posts. As ever, the F# language specification (section 6.3.10) is your friend.

Published on 10 Oct 2014 permalink Find me on Github

comments powered by Disqus