The chalk particles embed themselves into the small pores on the surface.
Although a chalkboard seems smooth to the touch, it is quite rough at the microscopic level, with pores that reach below the surface.
When you drag chalk across the board, friction causes small particles of chalk to rub off onto the surface.
If you leave the markings for a long time, some of the chalk particles will work their way into the pores.
A brush will remove the surface particles, but it will not be able to get at the particles in the pores.
