Streaming data

Streaming data#

  • Data that can be loaded chunk-wise (single or a few at the time).

  • Data flowing through a socket/pipe/streaming API/…

  • Typically requires:

    • Checking if stream is active.

    • A reactive reading mechanism or time delayed loop to check for updates.

    • Error/exception checking/handling.

    • A receiving enitity that can use the incomming data, e.g., printing to screen, updating graphics, updating a model, predicting some outcome/state/quantity, …

Live reading of Twitch chat#

# Check if user is logged in ("==" active chat)
import requests
user = "k3soju" # Change this to the user you want to check, e.g., epicdan22, zackrawrr, summit1g, 
# mizkif, cohhcarnage, k3soju, etc.
response = requests.get("https://decapi.me/twitch/uptime/"+user).text
is_online = response != user+" is offline"
print(is_online)

True

# Connect to the Twitch chat using River
from river import stream

if is_online:
    oauth = open('../../../No_sync/twitch_oauth','r').read()
    twitch_chat = stream.TwitchChatStream(
        nickname="khliland", # Exchange with your Twitch username
        token=oauth,
        channels=[user]
    )

# If the user is online, print the first 2 messages
if is_online:
    messages = 10
    for item in twitch_chat:
        print(item)
        if messages > 1:
            messages -= 1
        else:
            print("Puh, that's enough!")
            break

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 5, 61215), 'channel': 'k3soju', 'username': 'mazedmarky_ow', 'msg': 'FBtouchdown'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 5, 610087), 'channel': 'k3soju', 'username': 'backporch2', 'msg': 'ambessa emisary with 0 wins LUL'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 5, 985404), 'channel': 'k3soju', 'username': 'oceanicmission', 'msg': 'why did my Twitch3/6 Bruiser+BruiserMundo just lose to Conq. Gangplank?? wtf is GP'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 6, 625846), 'channel': 'k3soju', 'username': 'jzhutft', 'msg': 'he gave permission HandsUp'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 8, 703861), 'channel': 'k3soju', 'username': 'xkimchiho', 'msg': 'bottom right bud'}
{'dt': datetime.datetime(2024, 12, 4, 21, 21, 8, 882244), 'channel': 'k3soju', 'username': 'unfollow_fear', 'msg': 'NOW CHAT WILL START SAYING WHY GS BT DIFF'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 10, 41687), 'channel': 'k3soju', 'username': 'cl0uddem0n', 'msg': '@easyyou2 legit cant play the game without titans what are you on about'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 11, 175211), 'channel': 'k3soju', 'username': 'sukmedrysenpai', 'msg': 'stares at his TFT board, scratching his head. "Where’s my smeech?" he mutters, frantically dragging units around. Suddenly, someone in the lobby chimes in: "Bro, you look like a smeech. "Soju freezes. "Wait..." He looks at his reflection in the screen and sees it—round face, slightly derpy smile. He doesn\'t just need the smeech; he is the smeech. Without hesitation, he drags himself onto the board. "I AM the carry now," he declares. The board glows with newfound synergy. xdd'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 11, 556465), 'channel': 'k3soju', 'username': 'chonkerchris', 'msg': 'stares at his TFT board, scratching his head. "Where’s my smeech?" he mutters, frantically dragging units around. Suddenly, someone in the lobby chimes in: "Bro, you look like a smeech. "Soju freezes. "Wait..." He looks at his reflection in the screen and sees it—round face, slightly derpy smile. He doesn\'t just need the smeech; he is the smeech. Without hesitation, he drags himself onto the board. "I AM the carry now," he declares. The board glows with newfound synergy. xdd'}

{'dt': datetime.datetime(2024, 12, 4, 21, 21, 14, 73612), 'channel': 'k3soju', 'username': 'adileles', 'msg': 'FBtouchdown FBtouchdown FBtouchdown'}
Puh, that's enough!

Comments#

  • Here, River has formated everything nicely for us as dictionaries.

    • River’s streaming format is based on dictionaries to minimize overhead.

  • See escpecially the datetime formatting.

  • We will return to River and streaming Machine Learning later in the book.

Exercise#

Make a clock using World Time API

See also

Resources

Contents