HomeCompression of JSON data for URL Query Params
I've been working on a small web utility for analyzing costs associated with the operation of agricultural machinery.
It's a forms based thing where you enter a variety of information about costs and usage patterns.
The result is a summary of what the gear is costing you, both in total and per hectare.
Nothing you couldn't do in a spreadsheet (which is what I myself have previously done), just a bit quicker and easier.
A couple of things I wanted to achieve.
- Shareable results
- User configurable
- No backend
Sharing a link is as good a way as any of achieving goal 1.
Having no login and allowing all data fields to be editable takes care of goal 2.
Almost.
The more generic you make a tool, the more difficult it can be to use (ie., at some point we're back to the spreadsheet). So I wanted someone to be able to scaffold up an analysis and then share that with a colleague or client for further finessing.
To achieve this I use a configuration JSON object. This can be saved and reloaded by someone else: as a final result, or as a starting place for further work.
Normally with a web service the configuration is stored in the cloud and you can retrieve it from there. I couldn't do this unless I wanted to skip goal 3 above.
The solution I settled on was to encode the settings as a URL query parameter.
Here's an example link
And this is what the link actually looks like:
https://s4ag.com/lca/?config=N4IgbgpgTgzglgewHYgFwgAwDpsFYQA0IAJgIYAupaoAxqQA5yUA2AkkpDOQLYRLkxqIJhG6DUAbVBJSvNCEIgaCbvWR8BaKSBlz0AWVI0AFnCQRFYUswCuF1ABYAnC4xEkCchHEgAShAAzGygAIzgYYwhiAAIbJCYCaKDmZgBPaIR6ckRzYhAAXwJpWXsQADEoCDgAc2NyS2s7NCcANgw3HU9veQAFUihyaOV+UjNoRJgIUiTKmrqCop0S+VZVBAHo4hts72iAMmj2GHoIGmzkQSIrW3sAZlx29y6fACF14mho0iQYgHVjfoBaJlCDdfIAXSIH3olRocAoOSEulKAEEkEgbNZogBhBhMLEAEQgMNO8POSGiAAoAKQASkU3wx1gA+tDYWScsyTlAaBo0ABGDoeLw+AoQwogCAARxscHovH42IQXHEtAoEGq6zg3UkxT0IF+Uyg0T6A0uwi8Yi0etK2O22SQ1WiUWq3SIzDgAQgzIBAvaj3AjXsAHYHk8RfIAGr9bUwaIAdyYxmiMAQcGY0QBUGI5hgMAmxgQNmYMRCEEz-WIASMUSS62i3FIAA84NwbNxoh6vcdvgsbfIAKLMU7kKAID7mrve32oABM-quQYFDyFz3kABV4whYpMYhQvilotleIJxYtkfJ9OEaFhFCIrbqlvqAHIIABeb+Hk8906oc4XgY3Gg87hjqCjiuCEr9FMaLEAAEkWsBCAAmkaMAAPIBOh3IIsgSLLOgqH9HGCBApk0C4Sgi5AaguCgT466mHGzDIE6fBFrUAD8CwgERUBpL86wANZwSO0Gqk+pR8WkXyVNMlBQK69TUU0qD8qGq4RugPEEqMAnCaJZzifh+q6em6RblAQmyVMDQ0bOYadFp4FEGZ+lWQhwQSRe6BuRZwmZoh5rXKpAAc9HyJS5DbsQ27kJEmx6f5Vk2aQ9L5BK5FQAijpKiqQgADKkCEiE9FAcC8j00CeVAaAOHR5R2MwACqkxVVANXAQ1ZRNWVFX2PyWChhKkzkA61QScw3x5OgvpEDQwSVEgNCpEIygfPIKLNQSigwKk3Alcwm0ACQ8UEEBHegBVivkQA
Using large objects as query params is a bit sketchy. Google it if you want to understand why.
For this use case I was happy to take a crack.
My biggest concern was the length of the query string.
My approach:
- Stringify the JSON (remove all whitespace & line returns).
- Key compress the JSON keys.
- Compress the result (using lz-string and its 'compressToEncodedURIComponent' method)
I built a little utility to examine the results manually. You can see it here.
Below is the progression.
Step 1: Regular JSON (Length: 2348 chars)
{
"version": "0.0.5",
"data": {
"capitalInvestments": {
"items": [
{
"name": "",
"components": [
{
"name": "Machine",
"value": 49990,
"notes": "Refurbished unit, fully optioned"
},
{
"name": "Freight",
"value": 9600,
"notes": "Part container, sea freight"
},
{
"name": "Import duties & Inspections",
"value": 3500,
"notes": "Border and Wharf Fees"
}
],
"depreciation": {
"name": "Annual Capital Depreciation (%)",
"annual_depreciation_percent": 10,
"notes": ""
}
}
]
},
"equipmentCosts": {
"categories": [
{
"name": "Wear Parts",
"items": [
{
"name": "Cutting edges",
"life_ha": 10000,
"value": 750,
"notes": "Varies with soil hardness, should be hardfaced for maximum lifespan"
},
{
"name": "Electrodes",
"life_ha": 2000,
"value": 1500,
"notes": "Two used at all times"
}
]
},
{
"name": "Misc.",
"items": [
{
"name": "Nozzles",
"life_ha": 2000,
"value": 20,
"notes": ""
}
]
}
]
},
"areaAndHours": {
"YearsOfOperation": {
"name": "Years of operation",
"value": 5,
"notes": "How long do you expect to use the implement?"
},
"YearlyWorkHectares": {
"name": "Yearly area target",
"value": 1750,
"notes": ""
},
"DailyWorkHectares": {
"name": "Daily work area",
"value": 250,
"notes": ""
},
"DailyWorkHours": {
"name": "Daily work hours",
"value": 8,
"notes": ""
}
},
"operatingCosts": {
"LabourPricePerHour": 45,
"FuelUsePerHour": 25,
"FuelPrice": 1.75
},
"settings": {
"land": "ha",
"currency": {
"code": "AUD",
"symbol": "A$"
},
"fuel": "L"
}
}
}
Step 2: JSON after de-prettification (Length: 1280 chars)
{"version":"0.0.5","data":{"capitalInvestments":{"items":[{"name":"","components":[{"name":"Machine","value":49990,"notes":"Refurbished unit, fully optioned"},{"name":"Freight","value":9600,"notes":"Part container, sea freight"},{"name":"Import duties & Inspections","value":3500,"notes":"Border and Wharf Fees"}],"depreciation":{"name":"Annual Capital Depreciation (%)","annual_depreciation_percent":10,"notes":""}}]},"equipmentCosts":{"categories":[{"name":"Wear Parts","items":[{"name":"Cutting edges","life_ha":10000,"value":750,"notes":"Varies with soil hardness, should be hardfaced for maximum lifespan"},{"name":"Electrodes","life_ha":2000,"value":1500,"notes":"Two used at all times"}]},{"name":"Misc.","items":[{"name":"Nozzles","life_ha":2000,"value":20,"notes":""}]}]},"areaAndHours":{"YearsOfOperation":{"name":"Years of operation","value":5,"notes":"How long do you expect to use the implement?"},"YearlyWorkHectares":{"name":"Yearly area target","value":1750,"notes":""},"DailyWorkHectares":{"name":"Daily work area","value":250,"notes":""},"DailyWorkHours":{"name":"Daily work hours","value":8,"notes":""}},"operatingCosts":{"LabourPricePerHour":45,"FuelUsePerHour":25,"FuelPrice":1.75},"settings":{"land":"ha","currency":{"code":"AUD","symbol":"A$"},"fuel":"L"}}}
Step 3: Key compressed JSON (Length: 881 chars)
{"a":"0.0.5","b":{"c":{"d":[{"e":"","f":[{"e":"Machine","g":49990,"h":"Refurbished unit, fully optioned"},{"e":"Freight","g":9600,"h":"Part container, sea freight"},{"e":"Import duties & Inspections","g":3500,"h":"Border and Wharf Fees"}],"i":{"e":"Annual Capital Depreciation (%)","j":10,"h":""}}]},"k":{"l":[{"e":"Wear Parts","d":[{"e":"Cutting edges","m":10000,"g":750,"h":"Varies with soil hardness, should be hardfaced for maximum lifespan"},{"e":"Electrodes","m":2000,"g":1500,"h":"Two used at all times"}]},{"e":"Misc.","d":[{"e":"Nozzles","m":2000,"g":20,"h":""}]}]},"n":{"o":{"e":"Years of operation","g":5,"h":"How long do you expect to use the implement?"},"p":{"e":"Yearly area target","g":1750,"h":""},"q":{"e":"Daily work area","g":250,"h":""},"r":{"e":"Daily work hours","g":8,"h":""}},"s":{"t":45,"u":25,"v":1.75},"w":{"x":"ha","y":{"z":"AUD","aa":"A$"},"ab":"L"}}}
Note: if you are going to compress keys, you need to keep track of the schema (which makes changing formats a bit problematic).
{"a":"version","b":"data","c":"capitalInvestments","d":"items","e":"name","f":"components","g":"value","h":"notes","i":"depreciation","j":"annual_depreciation_percent","k":"equipmentCosts","l":"categories","m":"life_ha","n":"areaAndHours","o":"YearsOfOperation","p":"YearlyWorkHectares","q":"DailyWorkHectares","r":"DailyWorkHours","s":"operatingCosts","t":"LabourPricePerHour","u":"FuelUsePerHour","v":"FuelPrice","w":"settings","x":"land","y":"currency","z":"code","aa":"symbol","ab":"fuel"}
Step 4: Compressed and encoded (Length: 808 chars)
N4IghiBcIAwHTwKwgDQgEZVAYyyAJlANqgCmUIqIAZsWRQLJjYAWAlgHbloDmUALAE5hMNCwoAlUtQCuAJ3RsAzi1L4ABDI5sALinWyANoYCe6gPYAHHW3NdCAXxT1oAMTmk2PFjqp9IggBsMKIg4tAACmByOurYdjpgnKRy+kqkYAYeXj4gTi4gAJIAtpbmMer4MjakSuoAZOqFHEqWpNg2dkp+UADMiCFiFABC5fgp6mAcGgDqLNHU6q6ktXkAumhseOTQAIIcHDJghuoAwmCWusfqACKklh7YbGCdHOoAFACkAJRUAFZQACMoXClAcDjWThAAGs8IY6CAdiAZhk5OoojFumhCJASIiKKdqjYODx1GoeKs0MUgSFBiB-AB2AZDaAANWibFq6gA7roWOolOY2Cd5nJ8FwlEo0ixzDJDBp0KR1KL8NRmGoDOV1MUwAAPNjFGTFdSGNjUWqWKZ5Zz46AAUUM7R0cnM4yxIGpkAATLTeECBiCKAAVbnmTTpDQvSbGdQ2YqrCH5W0gBjKbBwKg4vFIgBy5gAXvnHe7PT66f4fSywZDIWgOHhzNsKABNVF1cyLKwpF62et+yCIKsACXM3JNdlJ+DDJllZN1bQ6sbDMnSsdU6gNlkd8Y4OgA-NaQJYm9BW9FTJMPJlEnIKb5+4CmYHoIeAI4nkA3JIX0NyaGXjIem9ZkwgoQ85A-L9hTMX9-xleR3X8AAOKs8ihbpIFAXxIH4QcQBkKAvTwgA3IE4CZKFuTwXUKHmKgTDwfMKF2ABVG4qDACA9gAEkPMBMGgAAZNCHCAA
How long before I realize this was a bad idea? Dunno. Time will tell 🤷