Can the rest of the chimney be removed?

Old house (1850), old fireplace, semi sealed, previous owner removed the chimney from the second floor up, I would like to remove the rest? I also see the cutout and replacement of section on floor where removed from the upper level. All advice greatly appreciated.