In my first GIS-course I was presented with the concept of quality measures for a georeferenciation. My lecturer taught me: “There it is: the RMSE or root mean square error. Keep in mind: it should be not bigger than half the pixel size!” That was all I got.
If you facing comparable words stick with us!
The RMSE is a distance measuring concept to compare expected with measured data. [Q] Where is the connection with georeferencing an image? [A] If you are doing a georeferenciation you are using a model to perform the transition from pixel-positions in your image to geographical coordinates. This model is calculating expected data.
As an example we will stick with an affine projection of an image.

pixel numbers and corresponding longitudes

An affine projection includes translation, compaction / expansion, rotation and combinations of these. To keep it more simple, we will concentrate now only on pixels in direction of x (longitudes):

the linear model behind

Of course the linear model or the connection between pixel numbers and longitudes is “already” defined by two points (if want to transfer this to x and y values: a affine transformation (polynomial transformation of 1st degree) of an image is defined by three points). But we are all producing errors: If you are using GCPs (Ground Control Points, like in an aerial photograph) or map coordinates: the data you are using is full of possible errors. To reduce your errors it is better to search and use more points and define their real-world coordinates. But doing so will result in another model. So you are changing your model which will try to fit to your data similar to a regression analysis:

changing the model by adding points

The best model will reduce the differences between your defined coordinate for a pixel and its calculated value according to the underlying model. This difference is calculated:
According to our example we will determine the difference between 40.7° and 40.2°:

detailed model

Due to the fact that the number of addends (or points used for the georeferenctiation) will influence the model most people tend to keep the number small. Nevertheless an increased number of reference points will provide a better model and can decrease the RMSE as well. It works the same way like the regression or correlation analysis: if the number of objects is small the model is saying nearly nothing.
An increased number of objects will increase model reliability and therefore quality of your work. In the process of the georeferenciation you will increase the reliability of your output coordinates in the map and your overall accuracy of your further work.