While research on retirement migration has primarily focused on retiree in-migration, there is little research on subsequent shifts away from such places after a period of residence. Traditional sea and tree change destinations are often framed in a positive light, where rural locations are celebrated for their attraction through aesthetic landscapes, perceived strong community values and lifestyles, alongside prospects of high-quality amenities and services. Smaller, non-traditional retirement destinations may still provide desirable characteristics, however the viability of such places later in life can become problematic. When migrating towards idealised landscapes, few retirees consider whether places are suitable to remain in, later in life. Once in place, retiree populations may be confronted with a lack of health and care facilities prompting difficult decisions about having to move again. This paper articulates the challenges of retiree migration to non-traditional retirement destinations, where services and facilities conducive for ageing in place are not adequate for long-term residence. Exploring two Western Australian rural case studies of Hopetoun and Kondinin, in-depth interviews with retirees highlight the complex decision-making around potential forced, secondary moves to more suitable locations. This two-step retirement migration poses issues, both in relation to the severing of long-term connections and place associations, alongside the difficulties associated with unplanned moves at such a late stage in life.