The historic fortress turned into the venue for pot smoker and alcoholics