How to handle streaming markdown returned from chatGPT?

Flow
1

Is there a way to stream the response from chatGPT and style the markdown?

I did see this post, Any simple MarkDown Viewer?, and I tried to implement something similar, but I couldn’t figure out how to handle the streaming correctly.

It seems like a Javascript component might be the way to go, but I’ve never done that from Flow.

It’s funny, with so many AI project examples out there, you’d think this would have been solved already. I did look at Marcus’ docs-assistant, but that’s a Hilla app.

Maybe a workaround is that I stream the text, then, after it’s done, go back and style it.

2

@Matti has a handy MarkdownMessage in his addon. If that isn’t exactly what you’re looking for, you can probably at least look at how he implemented it.

Here’s how I used it in one demo java-chat-with-documents/src/main/java/com/vaadin/demo/views/ChatView.java at d5f10f15c630ef6ec3bcb0021b458d804c797ea4 · marcushellberg/java-chat-with-documents · GitHub

2 Likes
3

Yes, I did exactly that for my minimal spring-ai-chat:

chatClient.stream(prompt)
    .doOnComplete(() -> chatHistory.add(new AssistantMessage(reply.getMarkdown())))
    .subscribe(cr -> reply.appendMarkdownAsync(cr.getResult().getOutput().getContent()));

The addition I needed to make to the original component was to get the streamed Markdown back (for chat history to work), but that is included in the library already.

1 Like
4

Thank you! I’ll try it out later today.

5

Does this way of doing it not block the UI when there are many lines ?

1 Like
6

It is not blocking the UI, but good that you mentioned… streaming only makes sense if you have enable websockets using @Push annotation in your application class.

This should get your streaming chat to work.

2 Likes
8

I wrote a blog post on the topic recently with a more generic markdown component than the MarkdownMessage above Displaying streaming markdown in Java and React UIs with Spring AI | Vaadin

1 Like
9

I’m streaming from open AI no problem. as the response comes in, I simply update the UI thread accordingly. observer pattern. And we can style it on the fly by adding everything as inner html. Maybe I don’t understand the question fully.

image
image3544×1320 342 KB

10

I hope you’re aware of the security implications of adding external content as inner html? I’d strongly recommend passing all incoming content through e.g. JSoup or OWASP HTML Sanitizer first.

1 Like
11

HI Olli

I know a little about Jsoup but this had not occurred to me.

Do you have any documentation ?

12

Here: Prevent cross site scripting with jsoup

1 Like