Hopscotch project JSON reader

[t1_hopscotch]

Hey Hops just completely for fun, i just made a small program to get the JSON from a project url and save it into a file. it’s a little pointless at the moment but was very interesting trying to work out how to try and do it, along with looking things up along the way

I would love to share the code, just at the moment it is very messy and makes no sense but if Hops are interested, would very very much love to try explain

import urllib.request as urllib_request
import os
import json
from lxml import html

def read_url(url_string):

    '''
    Takes a Hopscotch project URL (may need to have .html at the end, but unconfirmed) and writes its data into a .json file
    '''

    url_obj = urllib_request.urlopen(url_string).read()
    page = html.fromstring(url_obj)

    for div in page.xpath('//div'):

        # Get the JSON from the 'data' attribute of the <div> tag
        # This is made on the assumption that there is only one <div> tag that has a 'data' attribute where Hopscotch stores the JSON data
        div_data = div.get('data')

        if div_data:
            print('Project JSON was found.')

            # Convert the string containing the data from the <div> tag into a Python object
            project_data = json.loads(div_data)

            write_json_file(project_data)

def write_json_file(data):

    '''
    Takes a Python object and writes it into a .json file
    '''

    project_id = data['uuid']
    file_path_data = os.path.join('project_data', str(project_id) + '.json')

    with open(file_path_data, 'w') as project_file:
        json.dump(data, project_file, ensure_ascii=False, indent=4, sort_keys=True)
        print('Writing the JSON to a file.')

read_url('https://c.gethopscotch.com/p/ykcmajup8.html')

Please feel very free to ask about anything! I can imagine a first question might be:

What is JSON? (JavaScript Object Notation)
It is just a format for storing data in general, not a great first explanation but I can elaborate. There are mentions around the forum too about it if you can’t wait for me (searching ‘Hopscotch json’ brings up most stuff)

Can I try running the code?

Most certainly! If you are asking this question, of course I can show you :) and looks like I can go with making a tutorial now :P for the time being:

• you can go to the link and copy and paste the code into a text editor (if you are not sure about this, IDLE is where you can start)

• save it as a Python file somewhere (just add .py to the end of the file name)

• in the folder where you saved it, make another folder and call it project_data

• open the Python file again and go to the last line, where it says read_url('https://c.gethopscotch.com/p/ykcmajup8.html') but replace that link with any project link you like (you may have to add .html at the end there after you paste the project link in)
  (these are not very accessible instructions so I would like to update if anyone wants to try)

• run the project (if you are using IDLE, you can press F5)

• after a little wait, you can go to the project_data folder and a .json file will appear. this is the project JSON file. If you haven’t seen it before, hopefully later on maybe I can try point out a few things

• Oh dear I just remembered you might have to download some python libraries which is a bit extra, I will have to address that later. This is just an initial thing so sorry if this doesn’t work :(

---

I would like to thank all previous Hops and discussions on forum referring to the JSON files for projects it encouraged me to explore! I will update this if anyone is interested

P.S. If this looks a little bit like ‘ahhhh!’ please don’t feel like that(!) one of the reasons of me sharing this is so that you can get to see and have a chance to try understand a few more things too! And plus I am sharing this very quickly at the moment and not explaining much at all hence I’ve just made it even more confusing >_<

Also this is basically my first time peeking around, so if this isn’t a good idea (e.g. for any security purposes or anything, I did hear something about that), Hopscotch Team, please let me know

[Currency5097193]

What is json

[BuildASnowman]

Amazing job! The code is very neat, too. So, you have now made a project that can read JSON’s, I think the natural next step - one that can edit them

[Sweetlina]

Wow! That is very neat and organized! I can see everything and where everything is! Great joB, (It sounds so werid to say that cause you are much older than me… I shouldn’t have said that)

[anon68105100]

Wow, this is amazing! I love how organized the code is, great job! :D

[William04GamerA]

This is amazing! Great job @t1_hopscotch

[MR.GAM3R]

I’m going to try this out later. This looks sweet. Good job @t1_hopscotch!

[t1_hopscotch]

@Currency5097193

What is json

Whoops, should have added that it stands for JavaScript Object Notation, but that itself doesn’t explain much I will add that in though to clarify!

---

@BuildASnowman

Amazing job! The code is very neat, too. So, you have now made a project that can read JSON’s, I think the natural next step - one that can edit them

Wow thanks BuildASnowman You are one of the people I would like to thank! — I learnt from peeking around on the forum seeing what you’ve talked about regarding Hopscotch project JSONs

Hehe, I actually did have more plans that I didn’t mention — I was hoping/dreaming (dreaming, more like) to make something so that you could view the code blocks more easily from a non iOS device (for example, if you’re on a computer, when someone links a project, you might just want to see the blocks which would be enough to help with a problem/question or just see how an aspect works for example, but you can’t do that at the moment).

I actually didn’t think much about an editor (I thought it might be beyond me just at this point ) — I saw @Rawrbear and @K1ish were working on one using GameMaker, but now I think about it, that would be pretty… fun… I’d need to be more familiar with how the blocks/code is arranged in the JSON file, but I think I can sort of see how to try edit the data using Python…

(I recently got to be part of working on a website that uses Python for a server, so having it hosted as a website is a potential possibility… but like I just said, these are just little dreams at the moment but would be very, very fun to try and do )

---

@Sweetlina

Wow! That is very neat and organized! I can see everything and where everything is! Great joB, (It sounds so werid to say that cause you are much older than me… I shouldn’t have said that)

Ooh thank you Sweetlina; I thought it would be pretty messy and confusing I will still try to clear it up though I think And also, I think it’s lovely that you’re giving encouragement that’s awesome, no matter how old anyone is (also I don’t think ive really mentioned much about my age anyway actually )

Thank you @anon68105100 and @William04GamerA too

---

@MR.GAM3R

I’m going to try this out later. This looks sweet. Good job @t1_hopscotch!

Ooh I can go update the instructions (I shouldn’t wait for a good reason, but this is a great one!) And also thanks to you too, @MR.GAM3R — I learnt from seeing you talking about project JSONs around the forum too

I would like to add that like 90% of the code there was searched up basically; really! — it is awesome seeing that you can indeed learn as you go!

[RubyWolf1]

That’s really cool X3

[Stradyvarious]

That’s very clever. Great work.

[t1_hopscotch]

I will just jot down a few things here

You can edit this too, you know!

Ideas/investigations/possible todos :

• can save project to a file

• document more on Hopscotch project JSONs so more Hops can see

• possibly, see how going in another language might go

(Dreams) :

• render blocks in JSON files for viewing??
• edit JSON files??
• Discourse plugin??? so we can view directly on forum (very big dream)

Wonderings :

• Does the order of keys in dictionaries matter for Hopscotch reading a JSON file? (In Python, the key order for dictionaries is different every time you start it)
• What does the JSON for code look like if I put more blocks in a project? (e.g. have Ifs and Repeats inside each other, and Draw Trail and so on)
• general idea of structure of JSON file

‘Purposes’:

(this doesn’t really have a purpose — main is just for fun )

• Just for fun!
• So Hops can see more with project JSONs
• Viewing code for projects more conveniently —
  e.g. if someone has a question about why some code isn’t working as expected, or if you just want to See Code for a project while viewing in a browser and you don’t have an iOS device with you

“Can I get involved…?”

Yes!!

• @Rawrbear has ideas for a Hopscotch IDE (integrated development environment) where you can make and edit Hopscotch projects

Hopscotch project JSON reader

You know what, I'll learn Python instead. I haven't actually touched C++ too much, plus it'll make embedding your code easier. I made a GitHub repo just now to store the new project's code in. Maybe others would be willing to contribute as well

• Organising all these posts I’ve made
  (e.g. group together similar/related things, having a list where everything is, updating todo lists maybe — these are not what you have to do; I enjoy doing these too and I can do these myself, it is just if you like doing that sort of thing as well, then go for it )

• Styling for the blocks (when it comes to displaying the actual blocks, e.g. making the text for all the Movement blocks red etc. Will probably involve HTML & CSS, you can learn about those!

• And of course if you want to try some of the programming, you are most welcome to do so!

• And anything that you can dream of. This itself is a dream

Resources:

How to start building stuff for Discourse if you're newbie (like myself)

[t1_hopscotch]

Jotting down a few other things… This is more informational.

Again, please feel very free to edit.

How the program works:

A general text description:

get the html from a project url
look for <div> tag that has a `data` attribute
get the value of that `data` attribute (as a string)
convert to an object (or anything that you can manipulate)
save the object to a JSON file

How to view the project JSON in your web browser (!)

1. Go to project link and right click over the player:
   

   

2. Click Inspect Element (how to peek behind the scenes of any website, or, in other words, ‘how 2 become L33T H4x0r/a mad hacker’ like in this PyCon (Python conference) talk)
   
   I’m using Safari, but this functionality is common to pretty much most browsers at least. In Safari, I navigated to Elements to see the HTML (hypertext markup language — is code used to make websites and uses <tags> to wrap around different things)
   
   You’ll find the JSON in the data= part of a <div> somewhere, and you’ll see it in a reaally long line:
   

   

You can copy and paste that somewhere and try to separate it manually, or the project files that are saved using the program I put above will have that automatically done too. I think some people have text editors that can also collapse things automatically for them, which is very handy (but I don’t have one that does. I have one that colours text so I can see things a little clearer though, it is called Atom.)

What does the project JSON contain???

I am still exploring this too Some things you might see:

• ‘Hey, I can see some block names!’ Yep there are the names of the blocks, this is the data for storing the blocks in projects I believe Let’s take a look at some (or one or two).
  

  

[I may have to leave this til later, but you can see this is an example for `Grow by 10%` for now]

• Anything that is a generated ID will often be a really long sequence of characters. Generally, IDs need to be unique and the longer a string is, the more options you have for naming it.
  

  

(For example, if you have a one-letter long name, there are 26 letters in the English alphabet that you can choose for it. But if you extend it to 2 characters, now you have 26*26 = 676 options for names. You can imagine the more characters it becomes, the more options you’ll have. If you have a system that has heaps and heaps of objects, you’ll need pretty unique names.)

Also I think there are probably security reasons too for IDs seeming long and a little random. And there’ll be other reasons too. I’m mentioning this mainly for if you happen to be looking at them and going like ‘why is this so long??’ which would be totally okay by the way

I heard we can edit the JSON. What happens if I edit the JSON?

There is a way to put JSON projects back into your Drafts (and get them from your Drafts) using iTunes and connecting your device to a computer. If the editing is valid JSON for a Hopscotch project, the changes will be made in the project too. There are Hops who are more familiar with this (plus the Hopscotch Team too of course :)

I’m not exactly sure how it works in Hopscotch but I just know JSON is a way of storing data generally. This is from a veery limited understanding and my initial aim is just to make this more accessible so please feel very free to correct.

So I’m guessing: the JSON isn’t exactly the project itself, it is a way of storing what is in the project. You store that the user who made the project is t1 (for example) and that these particular blocks are in the project (e.g. Draw Trail, Move Forward 300 etc.) And when the editor opens a project, it looks at these notes (what blocks are in it, in what order, what objects you said were in there, and who made it etc) and then knows what to show or display.

if you start copying and pasting the JSON section for a Grow block or something, and repeat doing that in a row, logically I’m guessing you’ll end up with a whole bunch of Grow By blocks in your project rule, right? Well, you know what, let’s go find that out… coz we can do that (might have to try later though coz I will have to head off for now )

And with the JSON ‘not being the actual project itself’, peeking around, I can see that the filename for a project ends in .hopscotch, so you can sort of see the JSON specifies more of the contents of a project —the blocks and rules inside it, its objects, its creator etc.

[Rawrbear]

Is this code open source? I’d like to figure out a way to use this in our project, if possible. ^^

[t1_hopscotch]

Hi Rawrbear! Yes! it is the first link at the very top of the topic that is awesome, (dream would be to work with others) Feel very free to ask about anything

[MR.GAM3R]

Is Python a web language (meaning you can integrate it into websites)? Don’t know much about it, lol.

[treefrogstudios]

That’s so confusing and cool XD

[Rawrbear]

Ahh, thank you so much! I’ll try learning C++ as fast as possible to get the application out sooner (if school isn’t in the way too much).

[Rawrbear]

Yeah, you can import Python plugins into a website! You can also make/run Python PC applications and plugins with it if you install the language. ^^

[BuildASnowman]

Oh man - this is exactly what I hoped for when I published my hack. JSON’s are the key to hopscotch, and you are now the gatekeeper

I made a programming language that translates to Hopscotch jsons before, to be able to make projects on my computer. I know quite a bit about the structure of these files - if you ever have a question, don’t hesitate to ask

[t1_hopscotch]

@MR.GAM3R

Is Python a web language (meaning you can integrate it into websites)? Don’t know much about it, lol.

@Rawrbear

Yeah, you can import Python plugins into a website! You can also make/run Python PC applications and plugins with it if you install the language. ^^

Yeah you can use Python for a number of things (it is referred to as a ‘general purpose’ programming language)

@Rawrbear

Ahh, thank you so much! I’ll try learning C++ as fast as possible to get the application out sooner (if school isn’t in the way too much).

Wow are you using C++ for the JSON editor that you were working on? haha there’s no rush or anything I see you’re pretty keen about it though! and you must have quite a few things on your plate already too

@BuildASnowman

Oh man - this is exactly what I hoped for when I published my hack. JSON’s are the key to hopscotch, and you are now the gatekeeper

I made a programming language that translates to Hopscotch jsons before, to be able to make projects on my computer. I know quite a bit about the structure of these files - if you ever have a question, don’t hesitate to ask

Wow that’s awesome to hear and wow…

That is really cool!! A programming language, did you say? that is epic; how does it work? And that’s lovely; thank you so much, I think I’ll have quite a few questions already actually I think for the moment they’re things that I can investigate myself first for now (and will probably get a better idea if I look through closely first)

Oh, I have some of my wonderings in one of the posts above, actually I think I just need to get a general idea by peeking around more first.