Just recently the network automation folks witnessed a great library to be ported from Python to Go - scrapligo.
Been working on learning go a bit and have published scrapligo https://t.co/NDXQ6khxCr -- still a work in progress, but has been a fun learning experience! Check it out and let me know what ya think! 🤠— Carl Montanari (@carlrmontanari) May 19, 2021
For me personally this was a pivotal point because with scrapligo the Go-minded netengs can now automate their networks with a solid and performant library.
One of the things that scrapligo packs is, of course, the ability to reliably talk to the network devices using the same command line interface as a human would normally do. That means that scrapligo would send and receive the pieces of data that an operator would send/receive if they were connected with a terminal over SSH.
As you may very well be aware, the typical output that a network device produces for a given command is unstructured, meaning that it is not presented in a way that can be effortlessly parsed by a machine.
If we were to send
show system information command with scrapligo towards a Nokia SR OS device, we would have not be able to get, say, the device version right away, since the response is basically the unstructured blob of text as the program sees it.
What can we do about it?
In an ideal world, you would have stopped reading this post, because ALL your devices were equipped with some kind of programmatic interface that returns structured data. Like in the example above we connect to the SR OS node with scrapligo netconf subsystem and retrieve back a result of a NETCONF Get operation.
We then can run a query on this XML document we received and get the data out just nicely.
Unfortunately we are not yet there, we have thousands of access devices in Service Providers network which do not have any of the fancy interface. We have Enterprise networks running decade old gear. And we also live in a harsh world where even if the Network OS has one of those fancy interface, the level of information you can query via them is not on par with what you can do over CLI.
The next best thing is to leverage the device’s ability to present the output as JSON. Then you can capture this output over SSH and let your JSON parser to do it’s thing.
For example, on EOS every show command can be represented as a JSON blob:
and with this tiny
scrapligo program you can easily retrieve all the data from this output:
that produces the following output:
That approach is a decent alternative to a missing programmatic interface and sometimes is the best option. But, as it usually happens, it is not universal. Many Network OS’es can not emit JSON for any given command, if at all. That means we need to resort to the parsing of the unstructured data ourselves.
And we back to square 1, where we usually get after some reality check. That is where we need to parse the unstructured output ourselves and get the blob of text we receive from a device to be transformed to some data structure which we can use in a program.
Of course, we can simply use Regular Expressions or even brute characters matching a loop, but when dealing with lengthy outputs (usually a product of a
show command), we often resort to a framework that can simplify the parsing.
For quite a long time the TextFSM python library was the answer to that particular task and since then a huge amount of textfsm templates were written to parse all sorts of outputs from various devices.
Being a Go person myself I was wondering if TextFSM exists in Go, since once we have scrapli in Go, having a parsing library in Go was the key piece missing.
Let’s have a look how it is used within scrapligo. For that exercise we will take a
show system information output from Nokia SR OS and use a textfsm template to create a structured data out of it.
the original textFSM templates might need to be touched by you, since Go regexp differs from Python RE in some parts.
To parse the output we will get from the Nokia SR OS device I will create a file with the textfsm template for this output under
sysinfo.textfsm file name. Here is the template body:
Now when the template is there, we can write the following scrapligo + gotextfsm program:
Easy-peasy! All thanks to the textFSM integration that scrapligo recently added!
Regardless which network/vendor/consulancy firm you employed with, you won’t be able to avoid CLI parsing activities at all times. The legacy gear is out there, with no other management interface but SSH/SNMP.
Before scrapligo it was quite tedious (I’d claim not worth it even) to automate network activities over SSH. Now the module packs almost everything you need to efficiently get going and write some nice automation programs or CLI tools.