How I experimented with AI (and went straight for a cup of red wine)
Anthropic first, then ChatGPT
Last summer I was on holiday in the South of Spain. I was working on my application and decided to try what I felt could be an experiment with the newly available LLM models. I had just been granted access to a dev api key from Anthropic and I already had an OpenAI api key, so I could test the two models on the same novel approach.
TL;DR For some reasons, Anthropic’s Claude worked better in this specific use case
Since I was working on rendering network graphs in my application to analyze corporate relationships, I asked myself how would an LLM model perform in:
Extracting entities and relationships from an arbitrary text (well, I knew already, both LLM models performed well for entities extraction)
Once extracted the entities and the relationships, would they plot the network graph directly? What would they actually do?
Would it be straightforward to implement? Would it be usable and would be the output insightful? How would it compare to a deterministic network graph rendering?
I got the answer to my questions the same day. And I got more answers the day after.
What I did first was to choose an arbitrary text; in my case, that was a random newspaper’s article in Italian. Just to make sure entities could be recognized more easily, also for debugging purposes, I chose an article about a coffee company wanting to expand its business abroad.
I prepared a frontend, a graph component and serverless functions to call the api endpoints of the two LLMs. The api endpoints responded correctly, so I was ready to go to the second step: How could I make sure the response was correctly returned in a way that could be used by my graph component? My graph component uses the vis.js library and I needed to make sure that the response could be correctly understood in order to render the graph.
TL;DR It’s all about the prompt (and some cleaning of the response)
This is when things became interesting. I tried to “force” the LLMs to return a structure in a specific JSON format. After a few attempts, it turned out that Anthropic’s Claude was more easily able to return the response in the correct format. The key was to hardcode an example structure in the prompt, though it must be considered that the example must be simple and clear. I tried a few examples and I eventually noticed that both LLMs failed in returning a clean JSON because of their insistence in including polite introduction to the JSON: “Here is your JSON data:…”, “The data you asked is presented below”, and similar. This was unnerving.
Ultimately, in both cases, I hade to clean the response before passing it to the graph component, though I hoped the hardcoded prompt (my idea was to plot the graph with the click of a button) would suffice. It did not, so, cleaning the response turned out to be an essential step.
Once the response was cleaned, I was set. Passing the response to the vis.js library worked well and yes, LLMs like Anthropic’s Claude and ChatGPT can plot a network graph from arbitrary text.
Now, since copying pasting text is not my normal use case, I prepared a component to upload a PDF (and now also Excel) file and plot the network graph from an arbitrary PDF or Excel file.
This is the result with Anthropic’s Claude:
And below there is the result with OpenAI’s ChatGPT, using the same file. Notice that the first graph is not well rendered, but the second is, by simply clicking again on the “Get a graph” button.
I then iterated a bit, updating the models and allow interactivity with the documents (more than one document at a time can be graphed, up to 5 documents) and the graph, so that now I can not only get a network graph, but I can also ask the LLMs about the graph or request a subset of the information in graph format or also generate tables, which can be easily exported in Excel format and seamlessly pasted in a spreadsheet for further use.
And yes, after seeing the first graph nicely rendering I went straight for a cup of a good Spanish red wine.