Imagine your value(time) function was a sine function. The correct average rate of return is 0. However, depending on where you put your start and end, it could be largely positive or negative. In particular, with SF housing, if you looked at 1950 to 1998 you would see maybe a 2% annual gain. If you looked at 1950 to 2005 you would see a 5% annual gain. The problem is that the time span we're looking at is not very large compared to the magnitude of the noise. Take the ending price minus the starting price is super sensitive to the short term noise.
A much better way is to best-fit a line through the prices and take the slope of the line. This is way less sensitive to the edge effects. There are probably even better statistical ways, but it seems nobody does this even though it's totally obvious and everyone should do this.