JSON as a Native Tcl Value

Cyan Ogilvie
Ruby Lane, Inc.
cyan@rubylane.com

JSON

  • Has emerged as the de facto standard for structured data exchange, replacing XML
    • Much more readable than XML
    • Simpler structure makes native representation easier for most languages
    • Much simpler to correctly parse and generate

Types are a Problem When Importing JSON

{
    "foo": "is this a dict?",
    "bar": null
}
{
    "foo": ["is", "this", "a", "dict?"],
    "bar": ""
}
{
    "foo": {
        "is": "this",
        "a":  "dict?"
    },
    "bar": "null"
}
 ➜ 
foo {is this a dict?} bar {}
or 
foo {is this a dict?} bar null

Types are a Problem When Serializing JSON

{
    "pin": {
        "location": {
            "lat": 40.12,
            "lon": -71.34
        }
    }
}

When generating JSON from Tcl the missing type info must be supplied

  • huddle:
    HUDDLE {D {pin {D {location {D {lat {s $lat} lon {s $lon}}}}}}}
  • yalj-tcl:
    map_open string pin map_open string location map_open string lat number $lat string lat number $lon map_close map_close map_close
  • tcllib's json::dict2json guesses the types, but doesn't attempt to descend into nested dictionaries, so this example isn't possible
{
    "query": {
        "function_score": {
            "query": {
                "bool": {
                    "must": [
                        {
                            "function_score": {
                                "query": {
                                    "function_score": {
                                        "functions": [
                                            {
                                                "random_score": {
                                                    "seed": 33485047948
                                                }
                                            }
                                        ],
                                        "boost_mode": "replace"
                                    }
                                },
                                "functions": [
                                    {
                                        "script_score": {
                                            "lang": "expression",
                                            "params": {
                                                "logfactor": 1
                                            },
                                            "script": "1 + (sqrt(-1.57079632679 * ln(pow(1-(2*(_score*0.99 *0.5+0.5)-1), 2))) * ln(1+(1+doc['clickcomp_raw'].value)*min(1.1,doc['crf'].value)*logfactor)) * 1.0"
                                        }
                                    }
                                ],
                                "score_mode": "first",
                                "boost_mode": "replace"
                            }
                        }
                    ],
                    "should": [],
                    "must_not": []
                }
            },
            "filter": {
                "bool": {
                    "must": [
                        {
                            "term": {
                                "shoptype": "rubylane"
                            }
                        },
                        {
                            "terms": {
                                "itemstatus": [
                                    "retail",
                                    "sale",
                                    "reduced",
                                    "sold",
                                    "hold",
                                    "auction"
                                ]
                            }
                        },
                        {
                            "term": {
                                "itemdisplayarea": "shop"
                            }
                        },
                        {
                            "term": {
                                "shopstatus": "active"
                            }
                        },
                        {
                            "or": [
                                {
                                    "term": {
                                        "aip": true
                                    }
                                },
                                {
                                    "term": {
                                        "aip_grandfathered": true
                                    }
                                }
                            ]
                        }
                    ],
                    "should": [],
                    "must_not": [
                        {
                            "terms": {
                                "shopnickname": [
                                    "test",
                                    "karen",
                                    "jennifer",
                                    "rlshop",
                                    "dontletthishappen",
                                    "sampleshop",
                                    "rlwhatsthis",
                                    "rlwhatwasthis",
                                    "rubylane-sold",
                                    "rubylux-sold",
                                    "rubyplaza-sold",
                                    "rlreproshop",
                                    "rlgiftcardshop"
                                ]
                            }
                        },
                        {
                            "term": {
                                "itemnrpics": 0
                            }
                        }
                    ]
                }
            },
            "functions": [
                {
                    "filter": {
                        "term": {
                            "aip": true
                        }
                    },
                    "weight": 1.2
                },
                {
                    "exp": {
                        "starttime": {
                            "origin": "now+1w/d",
                            "scale": "1000d"
                        }
                    }
                },
                {
                    "filter": {
                        "terms": {
                            "lane": [
                                "jewelry",
                                "rxjewelry"
                            ]
                        }
                    },
                    "weight": 0.9
                }
            ]
        }
    },
    "post_filter": {
        "bool": {
            "must": [],
            "should": [],
            "must_not": []
        }
    },
    "sort": [
        "itemstatus_sort",
        "_score"
    ],
    "fields": [
        "_source"
    ],
    "size": 30,
    "from": 0,
    "aggregations": {
        "added_since": {
            "range": {
                "field": "starttime",
                "keyed": true,
                "ranges": [
                    {
                        "key": "Today",
                        "from": "now-24h/h"
                    },
                    {
                        "key": "This Week",
                        "from": "now-168h/h"
                    }
                ]
            }
        },
        "price": {
            "range": {
                "field": "priceusd",
                "keyed": true,
                "ranges": [
                    {
                        "key": "Under $25",
                        "to": 25
                    },
                    {
                        "key": "$25 - 49",
                        "from": 25,
                        "to": 50
					},
                    {
                        "key": "$50 - 99",
                        "from": 50,
                        "to": 100
                    },
                    {
                        "key": "$100 - 199",
                        "from": 100,
                        "to": 200
                    },
                    {
                        "key": "$200 - 499",
                        "from": 200,
                        "to": 500
                    },
                    {
                        "key": "$500 - 999",
                        "from": 500,
                        "to": 1000
                    },
                    {
                        "key": "$1,000 - 4,999",
                        "from": 1000,
                        "to": 5000
                    },
                    {
                        "key": "Over $5,000",
                        "from": 5000
                    }
                ]
            }
        },
        "layaway_available": {
            "terms": {
                "field": "layaway",
                "size": 0
            }
        },
        "itemstatus": {
            "terms": {
                "field": "itemstatus",
                "size": 0
            }
        },
        "by_lane": {
            "terms": {
                "field": "lane",
                "size": 0
            }
        }
    }
}

Types are Only a Problem if You Convert

[json] is to 
{
    "pin": {
        "location": {
            "lat": 40.12,
            "lon": -71.34
        }
    }
}
... as ...
[dict] is to 
pin {
    location {
        lat 40.12
        lon -71.34
    }
}

Templates for Serialization

{
    "pin": {
        "location": {
            "lat": 40.12,
            "lon": -71.34
        }
    }
}

rl_json uses templates to supply the missing type information:

set lat 40.12
set lon -71.34

json template {
    {
        "pin": {
            "location": {
                "lat": "~N:lat",
                "lon": "~N:lon"
            }
        }
    }
}

Template type prefixes:

~S: string
~N: number
~B: boolean
~J: JSON fragment
~T: JSON template
~L: literal - used to escape literal strings that would match the above

Working with JSON

set someval 0xFF

set res [json template {
    {
        "foo": "~S:someval",
        "bar": "~N:someval",
        "baz": [
            "~B:someval",
            "~S:otherval",
            "~L:~S:someval"
        ]
    }
}]

json set res baz end+1 $res    
json set res new {"entry"}
json unset res baz end baz end-1

puts [json pretty $res]
puts "baz: [json get $res baz 0]"
puts "is a [json get $res baz 0 ?type]"
 ➜ 
{
    "foo": "0xFF",
    "bar": 255,
    "baz": [
        true,
        null,
        "~S:someval",
        {
            "foo": "0xFF",
            "bar": 255,
            "baz": [
                true,
                "~S:someval"
            ]
        }
    ],
    "new": "entry"
}
baz: 1
is a boolean

Heresy!

Sometimes types can be useful

  • When the types are an intrinsic property of the value rather than implied by how it is accessed, the structure is unambiguous and so a pretty printer can be provided.
  • Since lists and dictionaries are then unambiguous, the key path can index into both: 
    set json {
    {
        "foo": "bar",
        "baz": ["str", 123, 123.4, true, false, null, {"inner": "target"}]
    }
}

json get $json baz end inner

dict get [lindex [dict get [json::json2dict $json] baz] end] inner

Future Work

  • json template currently produces a string result, but experience has shown that the result is very often manipulated further and so preserving it as a pure JSON ObjType would be a performance win
  • json foreach and json lmap need to be NRE enabled
  • manpage
  • It would be interesting to apply the deduplication scheme to Tcl_NewStringObj

Availability

Available on the Ruby Lane github page: https://www.github.com/RubyLane

Licensed under the same terms as the Tcl core