Recent advances in digital multimedia capture, transmission and storage has led us close to realising the MEMEX vision, in which much of an individual's personal information can be stored for retrieval at a later stage. These personal archives (Human Digital Memories (HDMs)) are constructed from a wide range of data sources across various media types. In this paper we examine the challenges of building, organising, browsing, searching and recommending from large (multi-million item per annum) visual HDMs. We present our findings based on an extensive period of HDM generation (2 years and 2 million images), illustrating the challenges of and architectural requirements for managing such archives.