I think those errors are both related to memory limits, though the second one appears to be some limitation on the maximum size of a `std::vector` (
http://www.cplusplus.com/reference/vector/vector/max_size/ ). You should check what the `max_size()` is on your system to see how large of a tensor can be allocated, since ITensor's use an `std::vector` as a storage.
Glancing through that paper, it is possible they are using
SU(2) symmetry, though they don't seem to give many details about their calculation. ITensor only makes use of abelian symmetries, so for the Hubbard model it can conserve the
U(1) spin projection symmetry and the
U(1) particle number. Conserving
SU(2) would lead to more sparse tensors, and therefore could allow one to reach much higher bond dimensions without reaching memory and time limits.
Alternatively, they may just be using a computer with a huge amount of RAM, or some other tricks like distributed memory. If you are interested in reproducing their results, could you reach out to the authors for more details on how they performed their calculation? I would be curious how they reached such large bond dimensions (those are larger bond dimensions than I am used to seeing for DMRG calculations, without extra "tricks" being used like the ones I mentioned above).