A hot air balloon rises in the first place because the air inside is hot, which rises. So, if the air inside is not heated, it fails to rise and falls. Warm air rises because the particles get excited and jiggle with the heat, and cause the air to take up more space. So, as the surrounding air is probably colder and denser, the less dense hot air rises up in the hot air balloon.
