They're your total hearts for all characters.
For every game (the total 14-day loop is considered a single game), you earn those hearts. Saying something that appeals to a character earns you a heart from them. Saying something that greatly displeases them breaks a heart from them.
After you've earned them, the game will save the information that, say, Yoosung gave you a heart on day 3.
You can use hearts to buy things hourglasses, the premium currency. Spending these golden hearts will not get rid of the saved information on who gave you the heart and how many you have from them!
On day 4, a route will be chosen for you depending on what character you have the most hearts for.